Image processing model

Abstract

To provide a method for training an image processing model.SOLUTION: A method includes: generating image explanations for images in an original training set of the images; determining at least one domain and / or class which is not sufficiently expressed in the original training set using at least one LLM; generating, based on the determination, at least one instruction for a third LLM to generate at least one text prompt using a second LLM; generating at least one text prompt for a text-image model using the third LLM based on the at least one instruction; generating at least one composite image using the text-image model based on the at least one text prompt; and generating a strengthening training set of the images for use in training an image processing ML model. The strengthening training set of the images includes the original training set of the images and the at least one composite image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to image processing models and their training, and in particular to computer-implemented methods, computer programs and information programming apparatus. [Background technology]

[0002] Image retrieval is the process of searching and locating images from a database of images. The query for an image retrieval task can be in the form of text or an image. When the query is an image, the process involves searching for images similar to the query image.

[0003] Deep Metric Learning (DML) is a component of some image retrieval systems in which models are trained to measure image similarity through an embedding space optimized by loss functions such as triplet, contrast, or angle loss. These loss functions are governed by convergence theorems, which ensure that the model learns to minimize intra-class variance and maximize inter-class variance. However, generalization remains a significant challenge, as DML models tend to overfit when trained or fine-tuned on limited datasets. Limited generalization can reduce the accuracy of DML models and make them susceptible to adversarial attacks. Summary of the Invention [Problem to be solved by the invention]

[0004] In view of the above, there is a need for improved methods for training image processing models. [Means for solving the problem]

[0005] According to an embodiment of a first aspect, there is disclosed herein a computer-implemented method, said method comprising: generating image descriptions for the images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; Includes.

[0006] Features associated with any aspect / embodiment may be applied to any other aspect / embodiment. [Brief explanation of the drawings]

[0007] By way of example, reference is made to the accompanying drawings in which: [Figure 1] This is a diagram useful for understanding image processing. [Figure 2] This is a diagram useful for understanding image processing. [Figure 3] FIG. 1 illustrates a system. [Figure 4] FIG. 1 illustrates modules of the system. [Figure 5] FIG. 1 illustrates modules of the system. [Figure 6] FIG. 1 illustrates the operation of a module. [Figure 7] FIG. 1 illustrates the operation of a module. [Figure 8] FIG. [Figure 9] FIG. [Figure 10] FIG. [Figure 11] FIG. [Figure 12] FIG. [Figure 13] FIG. [Figure 14] FIG. [Figure 15] FIG. [Figure 16] FIG. [Figure 17] FIG. [Figure 18] FIG. [Figure 19] FIG. [Figure 20] FIG. [Figure 21] FIG. [Figure 22] FIG. [Figure 23] FIG. DETAILED DESCRIPTION OF THE INVENTION

[0008] FIG. 1 provides an overview of a standard image retrieval framework useful for understanding the present disclosure. In the image retrieval framework, in step S12, metadata is calculated based on images stored in a database (image indexing). The output of step S12 includes a signature based on the images in the database. In step S14, metadata is calculated based on an input image that constitutes a query / request. The output of step S14 includes the signature of the input image. In step S16, a comparator compares the signature of the input image with multiple signatures of stored images to search for similar images among the images stored in the database. Here, "metadata calculation" refers to the extraction of embeddings using a DNN, and then the comparator calculates the similarity between the embedding of the database image and the embedding of the query image. In contrast, in the following description, "metadata" refers to auxiliary information about the data. The DNN and comparator function are trained using deep metric learning techniques. The process of searching for similar images (in this case, using deep metric learning) constitutes image retrieval.

[0009] Image retrieval can be thought of as the process of searching and locating digital images from a large database using a query. The query can be an image or text. In the exemplary overview of Figure 1, the query is an image. Image retrieval can use deep metric learning for image discovery.

[0010] Deep Metric Learning (DML) involves learning a function to reduce the distance in a continuous latent embedding space between pairs of similar inputs (images). Unlike classification systems, which assign discrete labels, DML models assign each image a location in a continuous embedding space.

[0011] Figure 2 provides an overview of the DML concept. In Figure 2, a deep neural network (DNN) receives images and outputs embeddings corresponding to the images. As shown in Figure 2, the DNN (which is a DML model) assigns each image a location in a continuous embedding space (the "discriminative feature embedding space" in Figure 2). In Figure 2, the DNN classifies images into classes A, B, C, and D based on their proximity to each other in the discriminative feature embedding space.

[0012] The effectiveness of a DML model depends on its generalizability property.

[0013] Generalizability: The ability of a model to perform well on new, never-before-seen data: This reflects its ability to capture universal patterns rather than memorizing specific training examples. The higher the generalization potential, the less likely it is to overfit. Less overfitting leads to better accuracy on clean data and adversarial robustness of image retrieval models.

[0014] Existing DML models for image retrieval suffer from limited generalizability. The lack of diverse data (lack of training information) is the main cause of limited generalizability. Limited generalizability leads to poor performance on clean data, poor out-of-variance adaptation, and vulnerability to adversarial attacks. Vulnerability to adversarial attacks is primarily due to the high sensitivity of DML to training data and the limited generalizability of DML models.

[0015] The implementation of the invention disclosed herein may be referred to as RobustRetrieval. RobustRetrieval stands for Robust Image Retrieval Leveraging a Combination of Large-Scale Vision and Language Models, and is a framework that represents the specific implementation disclosed herein. RobustRetrieval is a multimodal framework for improving the training process of image retrieval models. The framework automates the generation of synthetic data to address the scarcity of diversity and class and domain imbalance in training datasets. It creates representative data from the real world that enhances model generalization.

[0016] RobustRetrieVAL can be thought of as a framework for enhancing the generalizability of DML models by automating synthetic data augmentation in an LLM-guided environment using Large Vision Models. RobustRetrieVAL involves detecting weaknesses and training deficiencies in the training data and addressing them by generating targeted synthetic data.

[0017] 3 is a diagram of a system 300 according to a particular implementation of the present invention. System 300 can be considered a framework for representing a method, and components of system 300 can be considered modules. The modules can be implemented on a computer / device (e.g., as discussed with reference to FIG. 19).

[0018] System 300 includes an image-to-text model 31, a data insight generator 32, an augmented protocol selector (APS) 33, a prompt generator 34, a text-to-image model 35, an outlier removal and diversity control (ORDC) module 36, and a training feedback module 37. The output of system 300 is a trained model 40 (which may be considered part of system 300). System 300 may be considered an example of the RobustRetrieVAL framework.

[0019] As partially described above, the system 300 includes: The data insights generator 32 analyzes the transformed training data information in text space, simultaneously detecting class and domain distribution inconsistencies and other potential missing and extensible information, and guiding the subsequent pipeline for targeted synthetic data generation. That is, the data insights generator 32 extracts metadata for semantic pattern analysis and data weakness detection. The Augmentation Protocol Selector (APS) 33 dynamically defines the protocol for augmentation and ensures that the synthesized data is diverse and useful. In other words, the APS 33 strategizes and controls the enrichment of the data. · A text-image model35 for generating images from generated text descriptions (prompts). Next, the Outlier Removal and Diversity Control (ORDC) module 36 further refines the transformed image data against the text descriptions defined by the APS, aligning it with the distribution of the original dataset and maintaining semantic consistency. In other words, the ORDC module 36 is intended to optimize the quality of the synthetic data. · A training feedback module 37 for adjusting and updating the augmentation schema based on the filtering loss and model performance.

[0020] The operation of the modules of system 300 will now be described in more detail.

[0021] The original training data (sometimes called the original training set of images) is input to the image-text model 31 . The original training data, for example, in some implementations, is a standard image retrieval benchmark (Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011), Tech. Rep. No. CNS-TR-2011-001, California Institute of Technology; and / or Krause, J., Stark, M., Deng, J., & Fei-Fei, L. (2013), 3D Object Representations for Fine-Grained Categorization, In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) (pp. 1-7). Sydney, Australia; Song, HO, Xiang, Y., Jegelka, S., & Savarese, S. (2015), Deep metric learning via lifted structured feature embedding, CoRR abs / 1511.06452 (2015), arXiv preprint arXiv:1511.06452; Liu, Z., Luo, P., Qiu, S., Wang, X., & Tang, X. (2016, June), DeepFashion: Powering robust clothes recognition and retrieval with rich annotations, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)), or e-commerce product data images, for example.

[0022] The original training data may include metadata associated with the images, such as data labels and / or other metadata information (e.g., metadata information and / or label information available on the internet).

[0023] The image-text model 31 generates image descriptions / captions for images in the original training data. Image description generation can utilize state-of-the-art visual image captioning models (e.g., Li, J., Li, D., Savarese, S., & Hoi, S. (2023). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597) to generate text descriptions for the training images. The goal of the model 31 is to convert visual data into text insights compatible with later processing in the system 300. Image description generation can be viewed as image description extraction using the pre-trained image-text model 31. This extraction utilizes state-of-the-art large language models (LLMs) that nearly embody the assumptions of the Sampling Theorem to ensure that the discrete text representation maintains the continuous semantic consistency of the visual domain. The manipulation of the image-text model 31 can be performed by a data insight generator 32 .

[0024] Data Insights Generation Unit 32 This module employs a hybrid approach that integrates heuristics and LLM operations with prompt engineering to analyze image data (which is converted into text data), reducing token processing costs and addressing the limited context constraints of current LLMs.

[0025] The data insight generator 32 generates: Contextual analysis: Leveraging image descriptions generated from image-text models and available metadata. Class Distribution Insights: Check for class imbalance and guide the synthetic data generation process accordingly. Domain Distribution Insights: Analyzes image descriptions generated from image-text models to check for missing or imbalanced domains. · Additional Training Class: Leverage Prompt Engineering with LLM in available data class information. Data Summary Generation: Compiling a detailed summary of the dataset, identifying domain and class imbalances, and guiding subsequent data generation and augmentation (including for new classes) to enrich the training set for subsequent steps.

[0026] 3 shows the output of the data insights generator 32, which consists of 1) content summary, 2) domain imbalance, and 3) class imbalance. Domain and class imbalance may also include the identification of "new" domains and / or classes.

[0027] In other words, the Data Insight Generator (DIG) 32 efficiently processes visual data (or textual data, in cases where image description generation is performed outside of DIG 32) by combining heuristics and LLM with prompt engineering to extract concise, descriptive metadata in textual form. This module cleverly overcomes the contextual limitations often encountered in modern LLM, transforming data image descriptions into a rich, concise textual form for further analysis and revealing underlying semantic patterns to guide the synthetic data generation process.

[0028] The algorithm performed by DIG32 in a running implementation is shown below (with some explanations below each step):

number

[0029] In operation 1, DIG32 extracts descriptions of training images using a text-to-image model 31 (which can be considered part of DIG32).

[0030] Subsequent stages include tokenization in operation 2, cleansing in operation 3, extraction of nouns and verbs through information token extraction in operation 5, and identification of common terms C through frequency analysis in operation 6.

[0031] In operation 7, metadata is extracted from the original training data, including the metadata contained therein (e.g., labels and metadata from the Internet, for example). In operation 8, metadata contextualization R is created by inference of a pre-trained basic LLM on the metadata M. In operations 9 and 10, domain imbalance I_domain and class distribution I_class are inferred by the inference required by the LLM. This is tailored to the task context and purpose of both labeled and unlabeled scenarios. For example, in operation 9, salient domains are determined, and unrepresented and underrepresented domains are determined. Frequency analysis in operation 6 helps the LLM determine unrepresented and underrepresented domains. In operation 11, new class identification N identifies unrepresented classes, i.e., new classes, based on class imbalance (sometimes called class distribution) information and the LLM. In operation 12, context overview synthesis S_overview employs LLM prompt engineering to integrate I_domain, I_class, and N to provide actionable insights to the APS33. The context overview S_overview is sometimes called a summary statement. D_insight is generated by combining the context summary with I_domain, I_class, and N and formatting them for APS33.

[0032] In summary, DIG32 systematically analyzes image representations to generate nuanced contextual analysis, extracting insights into important class and domain distributions to mitigate imbalances and thereby enabling the generation of synthetic data to enhance model robustness. The operations performed by DIG32 are further described below with reference to the method steps.

[0033] An example output of the operation of the above algorithm performed by DIG32 is shown below.

[0034] Here is an example output from Operation 6 (Identify common terms using frequency analysis) running on a subset of the publicly available CUB-200-2011 dataset (Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The caltech-ucsd birds-200-2011 dataset (2011)):

number

[0035] The example output above shows the five most commonly occurring terms and their frequency.

[0036] Here is an example output of operation 7, working on a subset of the publicly available CUB-200-2011 dataset, when the original dataset contains label information:

number

[0037] Here is an example output of operation 8, running on a subset of the publicly available CUB-200-2011 dataset: Dataset name: Caltech-UCSD Birds-200-2011 (CUB-200-2011) Primary Uses: This dataset is intended for use in fine-grained visual classification tasks, and is particularly suitable for studies involving distinguishing between different bird species based on visual details. Release year: 2011 Species count: Contains images of 200 bird species, primarily from North America. Total images: 11,788 fully annotated images Notes provided: Each image contains a bounding box indicating the bird's location. To aid in the recognition of bird features (e.g., beak, head, wings), the locations of key parts have been annotated, and approximately 15 parts per species have been labeled. There are 312 binary attribute labels for each species that describe features such as color, pattern, and other species-specific characteristics. The images are labeled for species classification, the primary task for which the dataset was designed. Image characteristics: The images are in JPEG format and resolution varies across datasets. The images are not standardized in size and depict birds in their natural environment. Splitting the dataset: The training set contains 5,994 images. The test set contains 5,794 images. Author: This dataset was developed in collaboration between the California Institute of Technology and the University of California, San Diego.

[0038] Here is an example output from operation 9, running on a subset of the publicly available CUB-200-2011 dataset and using GPT-4LLM (e.g., OpenAI, R.: Gpt-4 technical report. arxiv 2303.08774): ChatGPT response: Based on the top stopping words and their occurrence, we present the following five key domains and the number of images associated with them. (1) Bird: The word “bird” appears 4,948 times. This shows that there are many bird-related images. (2) Nature: The word "branch" appears 2079 times, the word "tree" appears 1293 times, and the word "rock" appears 507 times. This shows that there are many images related to nature. (3) Water: The word “water” appears 1,134 times, and the word “swimming” appears 464 times. This shows that there are many images related to water scenes, such as lakes, rivers, and swimming. (4) Wild Animals: The word "sitting" occurs 3,029 times, and the word "standing" occurs 1,094 times. These words are not specific to a particular animal, but they indicate the presence of images of wild animals. (5) Flying: The word "flying" appears 592 times, suggesting the presence of images related to flying objects such as birds, airplanes, or other flying creatures.

[0039] The output I_class of operation 10 is, for example,

number

[0040] Example of target input and corresponding output N to the LLM in operation 11. Operate on a subset of the publicly available CUB-200-2011 dataset.

number

number

[0041] Here is an example of S_overview from operation 12, running on a subset of the publicly available CUB-200-2011 dataset:

number

[0042] The D_insight example in operation 13 configures a collection of related information in a JSON file.

number

[0043] Augmentation Protocol Selector (APS) 33 The APS 33 formulates enrichment objectives that are used by another LLM (Prompt Generator 34) to generate text prompts for creating controlled synthetic data via a text-image model. The APS 33 generates reference descriptions that provide downstream LLM class information necessary to create the enhanced text prompts. The APS 33 uses prompt engineering by the LLM that incorporates feedback from the Data Insight Generator 32 (and also feedback described below). Initially, the APS 33 relies solely on the output of the DIG to generate enrichment objectives for the prompt generation process.

[0044] That is, the APS 33 is responsible for generating reinforcement objectives and reference instructions that guide the downstream LLM (prompt generator 34) to generate text prompts for the downstream diffusion model. The APS 33 receives the D_insight output from the Data Insight Generator 32, the filtering feedback F_feedback, and the training feedback T_feedback. However, at the start of training, F_feedback=Φ and T_feedback=Φ (because there is no feedback available at this stage).

[0045] The algorithm performed by the APS33 in a running implementation is shown below (with some explanations below each step):

number

[0046] Function Description GenerateReferenceDescription: Generates a reference description that includes class information. GenerateAugmentationObjective: Generates an augmentation objective for downstream LLMs based on the reference description. IntegrateFeedback: Adjusts the reinforcement objective by incorporating feedback using LLM. In the first pass, T_Feedback and F_Feedback are not present (i.e., 0) and the reinforcement objective is generated purely based on the output of the Data Insights Generator component.

[0047] APS33 is a strategic module that defines the augmentation objectives for the downstream LLM (Prompt Generator 34), which is responsible for generating text prompts for synthetic data generation. APS33 leverages the analytical and inferential capabilities of the underlying LLM to translate data insights, filtering, and training feedback into actionable augmentation plans to guide the synthetic data generation process.

[0048] As described in the algorithm above, the APS 33 begins by interpreting the distilled insights D_insight from the Data Insights Generator 32. Next, the APS 33 systematically constructs a set of augmentation objectives E_objective and reference descriptions R_description, which serve to instruct another LLM responsible for generating text prompts for downstream synthetic image generation. These objectives are carefully tuned to address identified class imbalances, domain gaps, and to supplement the training data with new classes. This allows the resulting synthetic data to precisely target areas in the training dataset that require enhancement.

[0049] After the iterative per-class synthetic data objectives are used for the text prompt generation process and synthetic image generation, APS33 incorporates feedback from the model's performance and the data filtering stage (described below) into the subsequent training stage. This feedback helps adjust the augmentation objectives. This allows APS33 to adapt its synthetic data generation strategy to the model's evolving needs. Through its systematic and adaptive approach, APS33 plays a useful role in the RobustRetrieVAL framework, ensuring that synthetic data generation is both targeted and effective, leading to significant improvements in the training and performance of the model during training.

[0050] Here is an example of the E_objective output for a run running on a subset of the publicly available CUB-200-2011 dataset:

number

[0051] Here is an example of the R_description output from an example run, running on a subset of the publicly available CUB-200-2011 dataset. This example is for the class "Bald Eagle."

number

[0052] Prompt generation unit 34 This component uses the LLM to create text prompts for downstream text-image models 35 in the pipeline that meet the enhancement objectives provided by the APS 33. The text prompts are generated incorporating class information from ReferenceDescriptions received from the APS 33 and are guaranteed to meet the EnhancementObjective.

[0053] The generation of text prompts is sometimes called caption generation. Given an enhancement goal (or a single goal) from the APS 33, the LLM as a caption generation module (or prompt generator) 34 creates a collection of text prompts T_prompts using a set of reference class descriptions. For each class c∈C, the module converts the abstract enhancement goal into a concrete linguistic construct for image generation, generating a prompt t that is consistent with the enhancement strategy. c Construct ∈T_prompts.

[0054] 4 shows the function of the prompt generator 34 (herein labeled "LLM"). The LLM 34 receives augmentation goals and reference descriptions from the APS 33 and generates text prompts based on these. Reference descriptions are provided for each class. Multiple text prompts are generated for each class.

[0055] The example text prompts output from prompt generator 34 are examples run on a subset of the publicly available CUB-200-2011 dataset, along with the above examples of R_description and E_objective output.

number

[0056] Synthetic Image Generation In this phase, a pre-trained text-image model 35 synthesizes an image from the text prompt generated in the previous stage. Figure 5 shows the operation of the text-image model 35. The text-image model 35 receives the text prompt and outputs a (synthesized) image generated based on the text prompt. The text-image model 35 can be any (pre-trained) text-image model. Possible text-image models include: (1) Stable Diffusion 2 (Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition (pp. 10684-10695).), and (2)SDXL (Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Muller, J., ... & Rombach, R. (2023). Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952.)

[0057] That is, synthetic data generation can be thought of as occurring in two sequential phases: caption generation and image synthesis. The image synthesis process is driven by a pre-trained text-to-image model 35 that one-to-one maps generated text prompts T_prompts to a set of synthetic images I_synth. This mapping G is as follows: T_prompts → I_synth ensures that each synthetic image i∈I_synth visually embodies the text prompt t. This systematically augments the original dataset D with targeted synthetic instances through goals designed in distinct text spaces. This structured process ensures that the augmented dataset is precisely tailored to identified needs, expanding the breadth and depth of training data.

[0058] Outlier Removal and Diversity Control (ORDC) Module 36 The ORDC module 36 provides integrated outlier detection and diversity control for synthetic data refinement. It simultaneously addresses the challenges of outliers and diversity in synthetic data for image retrieval. The ORDC module 36 calculates the average distance to the class center within each original data training class and each original data training class. Outliers can be filtered out of the synthetic data by comparing the distance to the corresponding class center from each synthetic image with a threshold based on the corresponding average distance for that class. Diversity can be adjusted using a diversity factor (Δ). The diversity factor provides adjustable scaling for adaptively setting outlier and diversity thresholds.

[0059] The algorithm performed by the ORDC module 36 in a running implementation is shown below (with some explanations below each step).

number

[0060] The proposed ORDC module 36 is included to refine the synthetic dataset by removing outliers and controlling the diversity of the generated data. This synthetic data filtering process helps curate the most informative subset of synthetic instances to augment the training set of the image processing model being trained. Drawing indirect inspiration from reduced-reference image quality assessment techniques, particularly robust feature matching, the ORDC methodology leverages a pre-trained deep metric learning model to extract embeddings that encapsulate content-based features that are independent of image placement. That is, a deep metric learning model can be used to extract the embeddings O and S. The model used in the final trained image processing ML model can be used to extract these embeddings, or any other suitable embedding extraction model can be used.

[0061] For example, the algorithmic framework of ORDC module 36 uses a pre-trained deep metric learning model F to map the original and synthetic data Xo and Xs into feature-rich embedding spaces O and S, respectively, as described in the example algorithm above. In this way, ORDC module 36 can utilize embeddings from a model pre-trained specifically for metric learning tasks. These embeddings O and S enable content-aware comparisons that go beyond pixel-level differences and instead focus on the semantic similarity of the data.

[0062] Next, the ORDC module 36 calculates the o Calculate the class centroid C[l] and the average intra-class distance D[l] for X (operations 7 and 8) and use these to retain samples whose embedding distance to C[l] is within the range of D[l] scaled by the diversity coefficient Δ. s(Operations 11-14). This ensures that synthetic samples actively contribute to the ongoing image processing model's empirical risk minimization and improves dataset quality by balancing outlier rejection and diversity control. By dynamically adjusting the diversity factor, the ORDC module 36 can accommodate various levels of data complexity and is effective across diverse application domains. The method relies on readily available pre-trained models, eliminating the need for additional computational overhead and improving efficiency.

[0063] Thus, the ORDC module 36 performs content-aware synthetic data curation via semantic feature space analysis, dynamically adjusting Δ to accommodate variations in data complexity, and optimizing the distribution fit from the synthetic data to the original data without incurring significant additional computational cost.

[0064] The ORDC module 36 also performs a feedback process called a filtering feedback process to provide filtering feedback to the APS 33. That is, after generating the cleaned synthetic data X_clean, the ORDC module 36 verifies the consistency of the cleaned synthetic data Xclean with the enhancement objectives. s Through analysis of the change in class distribution before and after cleaning (X_clean) (excluding the predefined data-dependent exception scenario E_check), we prove that X_clean complies with the specified reinforcement objective. If an imbalance occurs, a compensation data generation signal F_feedback is issued to the APS 36 to correct the imbalance.

[0065] Feedback Filtering: Ensure that the data cleaned by the ORDC module 36 meets the overall Enhancement Objective. Imbalance detection: Identify class or domain imbalances introduced by ORDC cleaning. · Exception List Check: Cross-reference cleaned data with exception lists for compliance. ·APS Communications: Alert APS33 to mitigate imbalances. · Compensation Data Generation: Instructs the APS33 to generate additional data when an imbalance is detected.

[0066] The exception list check step mainly defines the stopping criteria to protect the APS from infinite loops. For example, the following algorithm operation can be used:

number

[0067] 6 is a diagram illustrating the feedback process performed by ORDC module 36, where checking for consistency with the enrichment objectives includes checking whether class imbalance has been resolved, whether domain imbalance has been resolved, and / or whether exceptions exist. The ORDC module does not check the synthetic data for new classes (i.e., classes not represented in the original training data).

[0068] The ORDC module 36 may be expected to use a simple heuristic to compare the classes of synthetic images before and after cleaning. For example, the ORDC module 36 may determine the number of synthetic images in a particular class and / or domain and compare this to the enhancement goal. For example, if the enhancement goal is to generate 50 images, but it is determined that there are only 20 images in the relevant class after cleaning, the signal from the ORDC module 36 to the APS 33 would be to generate 30 images in the given class. Because this could result in an infinite (or simply long) loop of processing, if the same enhancement goal is defined X times, the loop performed by the ORDC module 36 is stopped and the system proceeds to the next stage of processing.

[0069] Training Feedback Module 37 In this module, the filtered and corrected synthetic image data is added to the original training data, and an image processing ML model is trained and evaluated. The image processing ML model is trained using data including synthetic data generated using different diversity factors, and the best-performing model is selected. Figure 7 illustrates the training process, which includes merging the original training data with the filtered and corrected synthetic data for a given diversity factor, training an image retrieval model (the target image processing ML model), and outputting the trained model for the given diversity factor.

[0070] The weak class that generates the poorest performance of the best performing ML model is identified, and a signal is sent to the APS33 to generate more data in the identified weak class.

[0071] The performance of a model can be evaluated using the Recall@1 metric (or Recall@k, etc.). The Recall@1 metric can be understood in a simpler way as a kind of precision (at a high level). In more detail, this metric can be understood as follows: a query image is input, which is also contained in a database of images, and the image retrieval model whose performance is tested outputs an ordered list of similar images selected from the database. The Recall@1 metric checks whether the first image (the most similar image) is the same as the query image. In general, the Recall@k metric checks whether the query image is contained in the first k images of the ordered list. So, for example, the Recall@10 metric checks whether the input image is contained in the first 10 images of the ordered list. The Recall@1 metric is the most rigorous.

[0072] The algorithm (with some explanations below each step) executed by the training feedback module 37 in a running implementation is given below:

number

[0073] The function of checking the cleaned data to provide filtering feedback may be performed by the dynamic filtering module rather than by the ORDC module 36, or the dynamic filtering module may be considered to be included in the ORDC module 36.

[0074] The dynamic filtering and training feedback modules act as sentinels in the pipeline: filtering feedback (F_feedback) ensures that synthetic data adheres to initial reinforcement goals, while training feedback (T_feedback) identifies training weaknesses that require further attention, prompting additional iterative improvements in subsequent generations.

[0075] Based on the training feedback, APS33 updates the reinforcement goals and reference descriptions for underperforming (weak) classes. The new synthetic data generated from these goals is finally merged with the existing original training dataset and synthetic training dataset in the pipeline.

[0076] A target image processing ML model (e.g., an image retrieval model) is trained and / or fine-tuned using the synthetic training data generated within the pipeline and the original training data. That is, in the final phase, the refined synthetic data X_clean and additional refined data received in response to filtering and training feedback (X_ff and X_tf) are integrated with the original dataset as {X0, X_clean, X_ff, X_tf} to increase the diversity of the original training data and fix any identified gaps. The enriched dataset is then used to train and / or fine-tune the image retrieval model, improving its performance and generalization ability, especially in areas where it previously struggled.

[0077] The process for providing filtering feedback and training feedback and generating additional synthetic data based thereon may be repeated, for example, for a particular number of iterations, or until no inconsistencies and / or weak classes are identified. At each iteration, the process for providing filtering feedback is based only on newly generated synthetic images (and not, for example, previously generated and cleaned synthetic images).

[0078] Figure 8 illustrates a method according to an embodiment of the present invention. The method includes steps S31 to S36. It will be understood that the method steps may correspond to the processes and / or operations of the modules described with respect to Figure 3, and that the description of the method applies to the description of those modules, and vice versa. The method of Figure 8 may be considered to correspond to the RobustRetrieval framework.

[0079] Step S31 involves generating image descriptions, i.e., step S31 involves using the image-text model to generate image descriptions for the images in the original training set of images.

[0080] Step S32 includes determining under-represented and / or under-represented domains / classes, i.e., step S32 includes determining, based on the image descriptions, at least one domain and / or class that is under-represented and / or under-represented in the original training set using at least one large-scale language model (LLM).

[0081] Step S33 includes generating at least one instruction, i.e., step S33 includes using the second LLM to generate at least one instruction for a third LLM to generate at least one text prompt based on the determination of at least one domain and / or class (that is under-represented and / or under-represented in the original training set).

[0082] Step S34 includes generating at least one text prompt, i.e., step S34 includes generating, using the third LLM, at least one text prompt for the text-image model based on the at least one instruction.

[0083] Step S35 includes generating at least one composite image, i.e., step S35 includes generating at least one composite image based on at least one text prompt using a text-to-image model.

[0084] Step S36 includes generating an enhanced training set including at least one synthetic image. That is, step S36 includes generating an enhanced training set of images for use in training an image processing machine learning (ML) model. The enhanced training set of images includes the original training set of images and at least one synthetic image.

[0085] Step S31 can be considered to correspond, at least in part, to the operation of image-text model 31. Step S32 can be considered to correspond, at least in part, to the operation of DIG 32. Step S33 can be considered to correspond, at least in part, to the operation of APS 33. Step S34 can be considered to correspond, at least in part, to the operation of prompt generator 34. Step S35 can be considered to correspond, at least in part, to the operation of text-image model 35. At least some of the features described with respect to any of the modules in system 300 can be considered to be included in the corresponding method steps.

[0086] The step of determining at least one domain that is unrepresented or underrepresented in the original training set comprises: determining common terms in the image description; using the first LLM to infer, based on general terms, the domains represented in the original training set and the number of images in the original training set that represent each domain; determining that a domain represented in the original training set is under-represented if the number or percentage of images in the original training set that represent the domain is below a domain threshold; using a fourth LLM to determine whether there is at least one domain that is not represented by any image in the original training set (and if so, determining the at least one domain as the at least one unrepresented domain); It may further include:

[0087] The step of determining at least one class that is unrepresented or underrepresented in the original training set comprises: determining the number of images in the original training set associated with each class based on metadata and / or labels associated with the images; determining that a class represented in the original training set is under-represented if the number or percentage of images in the original training set representing that class is below a class threshold; using the sixth LLM to determine whether there is at least one class that is not represented by any of the images in the original training set (and if so, determining the at least one class as the at least one unrepresented class); may include:

[0088] Determining common terms among the image descriptions may include using natural language processing (NLP) techniques to determine common terms, such as tokenizing the image descriptions, removing stop words from the tokenized image descriptions to provide cleaned image description tokens, assigning parts of speech (POS) tags to the cleaned image description tokens, extracting nouns and verbs from the cleaned image description tokens based on the POS tags, and determining the n most frequently occurring nouns and verbs as common terms.

[0089] Each generated instruction may name an unrepresented or underrepresented class and / or domain and include the number of text prompts to be generated. Instructions may be generated for each such class. In some implementations, the instructions include, for example, enrichment objectives and reference descriptions, as described above.

[0090] In some implementations, the method further includes performing a cleaning process that may correspond to some of the operations of ORDC module 36. The cleaning process includes cleaning the composite set by removing any composite images that are determined to be outliers to generate a cleaned composite set of composite images.

[0091] The step of cleaning the composite set to generate a cleaned composite set may include, for example, generating a first embedding of the images in the original training set; generating a second embedding of the synthetic images in the synthetic set associated with the classes represented in the original training set; Calculating the mean embedding of each class of images in the original training set (based on the labels / label information); For each class of images in the original training set, calculate the average distance of the distances of the first embeddings of the images of the class from the mean embedding of the class; For a second embedding, comparing a distance between the second embedding and a mean embedding of the corresponding class to a class outlier threshold based on the mean distance of the corresponding class, and removing the synthetic image corresponding to the second embedding from the synthetic set if the distance is greater than the class outlier threshold; Includes.

[0092] The class outlier threshold for a given class comprises the average distance of the class multiplied by a diversity coefficient. A metric learning model may be used to generate the first and second embeddings, and / or a model used in the image processing ML model during training may be used. The embeddings are 1D vectors that all have the same size as each other (e.g., 128, 512, 2048, etc.). The average or mean embedding of multiple first embeddings is a vector of the same dimension, but with a simple element-wise arithmetic mean of those first embedding vectors. The distance may include any of Euclidean distance, dot product distance, and cosine distance.

[0093] In some implementations, the method includes a checking process that may be considered to correspond to the operation of the ORDC module 36 to provide filtering feedback. The checking process includes checking the cleaned composite set to determine whether additional composite images are needed, and if it is determined that additional composite images are needed, performing a cleaning compensation process. The cleaning compensation process may be considered to correspond to the operation of the APS 33, the prompt generator 34, and the text-image model 35 in response to the filtering feedback. The cleaning compensation process may include: generating, using the second LLM, at least one additional instruction for the third LLM to generate at least one text prompt; generating at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model; Includes.

[0094] Checking the cleaned composite set to determine whether additional composite images are needed includes, in embodiments, comparing the number of composite images in the cleaned composite set associated with each class and / or each domain with the number of text prompts specified for instructions, reference descriptions, or enhancement purposes corresponding to that class or domain, and determining that additional composite images are needed if it is determined that the number of composite images is less than the number of text prompts for each class and / or each domain.

[0095] Generating the at least one further instruction may include generating an enrichment objective including a context for the third LLM for use in generating the at least one text prompt, and generating at least one reference description, each reference description naming a class and / or domain, for example, naming a class determined to be unrepresented or underrepresented in the cleaned synthetic data.

[0096] As also shown in the description with reference to FIG. 3, in some embodiments, the method includes continuously repeating / iterating the cleaning, checking, and cleaning compensation process until the checking process determines that no additional composite images are required or until a check threshold number of iterations have been performed, and the enhanced training set includes the original training set of images and a cleaned composite set of composite images produced at each iteration of the cleaning process.

[0097] In some embodiments, the method includes a training feedback process that may be considered to correspond to the operation of the training feedback module 37. The training feedback process includes evaluating the performance of the trained image processing ML model to determine whether additional synthetic images are needed, and if it is determined that additional synthetic images are needed, performing a weak class compensation process. The weak class compensation process may be considered to correspond to the operation of the APS 33, the prompt generator 34, and the text-to-image model 35 in response to the training feedback. The weak class compensation process: generating, using the second LLM, at least one (further) additional instruction for the third LLM to generate at least one text prompt; generating at least one text prompt for the text-image model based on at least one (further) additional instruction using the third LLM; generating at least one (further) additional composite image based on at least one text prompt using the text-image model; Includes.

[0098] The training feedback process is training an image processing ML model using the enriched training set of images to generate a trained image processing ML model; Evaluating the performance of the trained image processing ML model using (multiple classes of) test images, and if the performance of the trained image processing ML model is below a performance threshold (e.g., T_perf) for any class of test images, determining that additional synthetic images are required and determining that class as at least one weak class; Includes.

[0099] The training feedback process further includes generating a plurality of enhanced training sets each corresponding to a plurality of diversity coefficients, each enhanced training set including images from the original training set and images from the composite set generated using the corresponding diversity coefficient. The training feedback process then includes separately training image processing ML models using the images from the enhanced training sets to generate a plurality of trained image processing ML models each corresponding to the images from the enhanced training sets, evaluating the performance of the plurality of trained image processing ML models to determine the trained image processing ML model with the best performance, and using the trained image processing ML model and the corresponding images from the enhanced training set in determining whether additional composite images are needed. For example, this is consistent with the operation of the training feedback module 37.

[0100] The method may include continuously repeating the training feedback process and the weak class compensation process until the training feedback process determines that no additional synthetic images are needed or until a training threshold number of iterations have been performed, whereupon the enhanced training set includes the original training set of images and the additional synthetic images generated in each iteration of the weak class correction process (synthetic images generated in the cleaning compensation process and images generated by the first synthesis with image removal performed in the cleaning process).

[0101] The Recall@k metric can be used to evaluate the performance of a model, for example as described above. It will be appreciated that there are many metrics that can be used.

[0102] The method may include training an image processing ML model using an enriched training set of images. The method may further include using the image processing ML model after training for at least one image processing task. The image processing ML model may include, for example, an image retrieval model using deep metric learning.

[0103] Figure 9 illustrates a method according to an embodiment of the present invention. The method includes steps S51-S64. It will be understood that the method steps may correspond to the processes and / or operations of the modules described with respect to Figure 3, and that the method descriptions apply to those module descriptions, and vice versa. The method of Figure 9 may be considered to correspond to the RobustRetrieVAL framework. Furthermore, the description of the method steps in Figure 8 may apply to the method steps in Figure 9, and vice versa.

[0104] Step S51 involves generating image descriptions, i.e., step S51 involves using the image-text model to generate image descriptions for images in the original training set of images, referred to as "original training images," that were input into step S51 of Figure 9.

[0105] Step S52 includes data insight generation, e.g., as described with reference to the DIG 32. Step S53 includes operation of the APS 33 to output instructions using the LLM and prompt engineering. Step S54 includes generating image descriptions using the LLM based on the instructions from step S53, e.g., as described with reference to the prompt generator 34. Step S55 includes generating synthetic data using a text-image model, e.g., as described with reference to the text-image model 35. Step S56 includes outlier removal and diversity control, e.g., as described with reference to the ORDC module 36. Step S57 includes filtering feedback generation, e.g., as described with reference to the ORDC module 36. Step S58 includes issuing new instructions for image description generation in response to the filtering feedback, i.e., updating the augmentation protocol for cleaning compensation, e.g., as described with reference to the ODC module 36 and the APS 33. Step S59 includes augmenting the original training data with synthetic data. This step may include generating a further synthetic image based on the new instructions for cleaning compensation, and generating a synthetic image based thereon.

[0106] Step S60 involves training a target image retrieval model (or image processing ML model) on the original + synthetic data. Step S61 involves generating training feedback, which includes identifying classes of data with low recall@1 scores. Steps S60 and S61 may involve the operation of training feedback module 37. Step S62 involves generating new instructions for image description generation in response to the training feedback, i.e., updating the augmentation protocol according to the training feedback, e.g., as described with reference to APS 33.

[0107] Step S61 involves augmenting the training data (including the original training data and the previous synthetic data) with newly generated synthetic data. This step may include generating the new synthetic data by generating image descriptions based on the new instructions and generating synthetic images based thereon. Step S64 involves training a target image retrieval model (or image processing ML model) using the final augmented training set of original and synthetic images.

[0108] It should be noted that at least one instance of each of the checking, cleaning, and cleaning compensation processes may be performed after generating a synthetic image in response to the training feedback process. It will be appreciated that the checking, cleaning, cleaning compensation, training feedback, and weak class compensation processes may be repeated until it is determined that no more synthetic images are needed and / or until a threshold number of iterations of the training feedback and / or cleaning compensation processes is reached.

[0109] The above descriptions of Figures 3-9 refer to LLMs. Each instance of an LLM may be different from all other instances. Alternatively, the same LLM may be used in several steps / modules. For example, an LLM may be trained to perform all of the tasks listed above (similar to an LLM like GPT-4). In this case, the same LLM may be used in all steps / operations. The difference lies in prompt engineering, which is responsible for setting individual goals for each task. It is also possible to assign different specialized LLMs, each specially trained, to specific steps / operations in the pipeline.

[0110] Below is a first set of examples / tests to evaluate the effectiveness of the proposed implementation.

[0111] Zero-shot learning test The proposed implementation, along with the RobustRetrieVAL framework, was tested using zero-shot learning, i.e., the training and test classes are separated.

number

[0112] DINO Reference: Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., & Joulin, A. (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE / CVF international conference on computer vision (pp. 9650-9660). ViT-S Reference: Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., ... & Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

[0113] The results of the zero-shot learning test are shown in Figure 10. DINO and ViT-S were trained with various diversity factors using the RobustRetrieVAL framework and compared to the baselines for these models. In the baseline, the models were trained using a hyperbolic loss function (Ermolov, A., Mirvakhabova, L., Khrulkov, V., Sebe, N., & Oseledets, I. (2022). Hyperbolic vision transformers: Combining improvements in metric learning. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 7409-7419)). The performance of the trained models was evaluated using the Recall@1 metric, which is shown in Figure 10. It is clear that for a diversity factor of Δ=1, the RobustRetrieVAL framework outperforms the state-of-the-art training of the hyperbolic ViT model.

[0114] Data-starved zero-shot learning test We tested the proposed implementation along the RobustRetrieVAL framework using zero-shot learning (i.e., the training and test classes are separated) similar to the above example / test, but with less training data than the above test. Training data preparation: Dtrain', containing classes Ctrain' and k images for each class, was uniformly sampled from the training data Dtrain in the above example / test.

number

[0115] The test data, ViT model, and evaluation process are the same as in the zero-shot learning example / test above. The results are shown in Figure 11, in sections titled "Training with Original Data Only" and "Training with Mixed Data." The values ​​in the "Training with Original Data Only" section are the performance metrics for models DINO and ViT-S trained using the traditional (state-of-the-art) method with k = 3 and k = 9 images per class in the training data. The values ​​in the "Training with Mixed Data" section are the performance metrics for models DINO and ViT-S trained according to the RobustRetrieval framework with Δ = 1 and k = 3 and k = 9 images per class in the training data. It is clear that the RobustRetrieval framework improves the performance of image retrieval algorithms under data-scarce conditions in no-shot learning scenarios.

[0116] Training without using the original training data Test We tested the proposed implementation along the RobustRetrieVAL framework using data-less training, i.e., only synthetic data was used for training (hence, k=0 in the above examples / tests, using the notation k=images per class from the original training data). Such tests may represent the performance of a trained image processing ML model on real data in a no-shot learning setting. Data-less training enables experimentation in settings where real data is limited, costly, or ethically challenging, paving the way for rapid prototyping and deployment by skipping the time-consuming data collection phase. Furthermore, data security and privacy are inherently easier to manage, and data-less training allows for controlled experiments targeting specific phenomena or corner cases.

[0117] The results of training without the original training data testing are shown in the "Training without data" section of Figure 11. The values ​​in the "Training without data" section are the performance metrics (same as in the tests above) for models DINO and ViT-S trained according to the RobustRetrieVAL framework. The number of images per class in the training data is k=0 (no original training data), Δ=1, 1.5, 3, and 5. It is clear that the RobustRetrieVAL framework enables training of zero-shot image acquisition models without access to the original training dataset.

[0118] Adversarial Robustness Evaluation The adversarial robustness of the DINO model trained using the RobustRetrieVAL framework was tested under the following conditions: Architecture: DINO Data: synthetic data cleaned with Δ=5 Data Phase: Training Attack method: Feature space PGD (projected gradient descent) Attack target: Evade Avoidance Targets: 1

[0119] For completeness, the following information is provided: "Data Phase": The "Data Phase" entry indicates whether the data used to create the adversarial attack was part of training the DML model or was test data. Feature-space PGD (projected gradient descent) attack: This is an adversarial optimization with an adversarial loss that operates on the feature vectors or embedding vectors of a deep metric learning model. In other words, feature-space PGD means that the attack is optimized using an adversarial loss on embeddings rather than labels in a traditional attack generation setting. For more information on adversarial attack methods, see: PGD Attack: Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017). Towards deep learning models resistant to adversarial attacks, arXiv preprint arXiv:1706.06083; Feature Space PGD attack: Rozsa, A., Gunther, M., & Boult, T. E. (2017, October), LOTS about attacking deep features, In 2017 IEEE International Joint Conference on Biometrics (IJCB) (pp. 168-176), IEEE; Singh, I., Momiyama, S., Kakizaki, K., & Araki, T. (2021, September), On brightness agnostic adversarial examples against face recognition systems, In 2021 International Conference of the Biometrics Special Interest Group (BIOSIG) (pp. 1-5), IEEE. "Evasion": The original image is perturbed by adversarial optimization so that it is no longer identified with its original category, "evading" the detection of the actual class. "Avoidance targets": The number of images that have the same identity as the image being used to create the adversarial attack.

[0120] Figure 12 shows the results of the adversarial robustness evaluation under the above conditions. Figure 13 shows the results under the same conditions, except for different values ​​of ε. ε is the L-infinity constraint on the size of the adversarial noise. The smaller this value, the less visible the adversarial noise is to the naked eye. Each ε value in the table represents the percentage of the adjusted original image pixel range. Figure 14 shows the results of the adversarial robustness evaluation under the same conditions as the results in Figure 13. However, the data used is the CUB-2002011 data (Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011), The Caltech-UCSD birds-200-2011 dataset), and the data phase is the test phase instead of the training phase.

[0121] The results of the adversarial robustness evaluation show that the RobustRetrieVAL framework improves the robustness of the VisionTransformer model (DINO) for image retrieval tasks. For synthetic data, RobustRetrieVAL achieves relatively high adversarial robustness. The improvement in adversarial robustness is especially high for highly imperceptible adversarial noise. Improvements in adversarial robustness begin to dry up as attack strength increases significantly and adversarial noise becomes more perceptible. Even with clean data, there is a moderate improvement in robustness due to RobustRetrieVAL. - In the case of clean data where the attack strength is excessively high, there is no improvement in robustness.

[0122] Class Imbalance Test To evaluate real-world scenarios where the number of training samples for the selected class set is limited, indicating imbalance in the training data, we prepared a subset of the CUB dataset (Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The Caltech-UCSD birds-200-2011 dataset). This involves limiting the number of training images to two each in predefined sets of 75 and 100, using 150 classes out of a total of 200 classes. Experiments were conducted using the DINO and ViT-S16 Vision Transformer models in a full-shot setting. The test set was kept constant, consisting of images not included in the training set, to ensure fair evaluation.

[0123] The results of the class imbalance tests are shown in Figures 15 and 16, which include graphs showing the metrics Recall@1 and Recall@2 versus the number of imbalanced classes in the training data. Figure 15 shows the results using DINO as the ViT model for image retrieval. Figure 16 shows the results using ViT-S as the ViT model for image retrieval. The "Baseline" values ​​were obtained using an image retrieval model trained in a conventional manner, and the "Implementation" values ​​were obtained using an image retrieval model trained using the RobustRetrieVAL framework with Δ=1. It is clear that in all cases, the models trained using the RobustRetrieVAL framework outperform the models trained in a conventional manner.

[0124] Domain Imbalance Test To evaluate practical situations where availability is limited in certain domains during training, indicating domain imbalance, we used the RobustRetrieVAL pipeline (i.e., the output of DIG32 processing) to classify the entire CUB (Wah, C., Branson, S., Welinder, P., Perona, P., & Belongie, S. (2011). The Caltech-UCSD birds-200-2011 dataset) training data into three domains: flying, sitting, and swimming. We then trained a Vision Transformer model using training data from each of those domains (but not the other two) (using the traditional method to obtain baseline results and RobustRetrieVAL for the "implemented" results). RobustRetrieVAL enhancements were implemented to target the underserved domains for each bird species. In our experiments, we used the DINO and ViT-S16 Vision Transformer models in a full-shot setting. The test set consisted of images not included in the training set and was kept constant to ensure fair evaluation.

[0125] The results of the domain imbalance test are shown in Figures 17 and 18. These include graphs showing the Recall@1 and Recall@2 metrics for each domain. The x-axis of the results represents discrete values ​​corresponding to the domains identified in the original training set. For example, the domain shown on the x-axis was the only domain in the training data from the original set of training data used for the model at that data point (i.e., two other domains were missing in that training subset). RobustRetrieVAL identifies the missing domain and performs targeted synthetic data augmentation focused on the missing domain. Thus, a single x-axis point represents a single domain present in the training data subset. A baseline model is trained on that data, RobustRetrieVAL performs synthetic data augmentation, and then retrains the same model to improve performance. Figure 17 shows the results using DINO as the ViT model for image retrieval. Figure 18 shows the results using ViT-S as the ViT model for image retrieval. The "Baseline" values ​​were obtained using an image retrieval model trained in a conventional manner, and the "Implementation" values ​​were obtained using an image retrieval model trained using the RobustRetrieVAL framework with Δ=1. It is clear that in all cases, models trained using the RobustRetrieVAL framework outperform models trained with traditional methods.

[0126] Considering the above experiments, the following becomes clear: · Performance improvements in zero-shot deep metric learning: RobustRetrieVAL demonstrated up to 1% performance improvement over existing state-of-the-art training of ViT models in traditional zero-shot deep metric learning scenarios. Benefits in low-data environments: When the original training data is limited, RobustRetrieVAL shows a performance gain of up to 2.3% compared to state-of-the-art training of the ViT model. Enhanced adversarial robustness: Models trained with the RobustRetrieVAL method showed a 1.9% improvement in adversarial robustness, indicating that image retrieval models that use RobustRetrieVAL in training are more resilient to adversarial attacks compared to those trained with traditional methods. Improvements for Imbalanced Input Data: In scenarios with imbalanced input data, RobustRetrieVAL improves performance by up to 5.24%, demonstrating its ability to effectively handle datasets with uneven class distributions. Augmenting target domains for incomplete data: For input data characterized by incomplete or missing domains, augmenting target domains in RobustRetrieVAL improved performance by up to 5.93%.

[0127] Below is a second set of examples / tests to evaluate the effectiveness of the proposed implementation. This second set of examples / tests uses an implementation of the RobustRetrieval framework with different parameters than the first set of examples / tests, and in this second set the implementation was tested on more data. The results in Figure 10 can be considered a subset of the results in Figure 22.

[0128] In these experiments, we rigorously evaluate RobustRetrieVAL in sparse data environments, focusing on the domain and class content-specific rarity of the available data. We then validate its generalization improvement capabilities on standard balanced benchmarks. In all cases, RobustRetrieVAL outperforms state-of-the-art (SotA) models.

[0129] Datasets and evaluation metrics We conducted experiments on three image retrieval benchmark datasets: CUB-200-2011, Cars196 (Krause, J., et al.: 3d object representations for fine-grained categorization, In: Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561 (2013)), and Stanford Online Products (SOP) (Oh Song et al.: Deep metric learning via lifted structured feature embedding, In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4004-4012 (2016)). As an evaluation metric, we adopted R@K, a well-known choice for evaluating image retrieval performance.

[0130] Baseline and Implementation RobustRetrieVAL is a current SotA, i.e., hyperbolic vision transformer DINO, for content-based image retrieval (CBIR) at 224x224 input resolution. H and ViT HWe evaluated the performance gain in data-poor scenarios compared to Additionally, a comprehensive set of baselines is considered for standard benchmark comparison, including Margin (Wu, C.Y., et al.: Sampling matters in deep embedding learning, In: Proceedings of the IEEE international conference on computer vision. pp. 2840-2848 (2017)), NSoftmax (Zhai, A., Wu, H.Y.: Classification is a strong baseline for deep metric learning, arXiv preprint arXiv:1811.12649 (2018)), MIC (Roth, K. et al.: Mic: Mining interclass characteristics for improved metric learning, In: Proceedings of the IEEE / CVF International Conference on Computer Vision. pp. 8000-8009 (2019)), and IRTR (El-Nouby, A., et al.: Training vision transformers for image retrieval, arXiv preprint arXiv:2102.05644 (2021)). The hyperbolic vision transformer includes a hyperbolic loss function to fine-tune the original ViT, DINO, and DeiT (Touvron, H., et al.: Training data-efficient image transformers & distillation through attention, In: International conference on machine learning. pp. 10347-10357. PMLR (2021)) models. The models in all experiments have ImageNet pre-training initialization and operate with an embedding dimension of 128.BLIP-2 (Li, J., et al.: Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, arXiv preprint arXiv:2301.12597 (2023)) was used to convert images to text metadata, and SDXL (Podell, D., et al.: Sdxl: Improving latent diffusion models for high-resolution image synthesis, arXiv preprint arXiv:2307.01952 (2023)) was used to synthesize realistic images from text descriptions. GPT-4 was used as the LLM inference engine. For ORDC filtering, the following hyperparameters were set to limit training complexity:

number

[0131] Data Preparation To evaluate RobustRetrieVAL for image retrieval, we employed the standard DML benchmarks CUB-200-2011, Cars196, and SOP, which originally have relatively balanced class and domain distributions. To simulate data scarcity and distribution bias, training subsets were designed by selectively editing specific patterns. Meanwhile, the original test set was kept unchanged to ensure an unbiased benchmark. Curating training subsets with controlled class distributions C is methodologically straightforward because benchmarks are widely used in standard supervised DML model training. However, established methods or standards for domain-specific bias and pattern omission have yet to be developed. To curate training sets using domain control, we used DIG to draw the dominant primary domains Dset within each dataset.

[0132] In the CUB-200-2011 dataset, domains were classified based on bird behaviors such as "sitting" (As), "swimming" (Aa), and "flying" (Af). Following a similar approach for the Cars196 data, domains were classified based on vehicle body type: "sedan" (Bs), "SUV-crossover" (Bc), and "sport performance or convertible" (Bp). To split the training data to induce domain imbalance, we defined the following domain imbalance creation function:

number

number

[0133] At the same time, a skewness parameter was used to introduce class imbalance:

number

number

number

[0134] Combining κ and ε allows us to construct training subsets that accurately reflect the idiosyncratic skewness that is typical in real-world datasets. To maintain consistency in our evaluation, we adhered to a standard 50:50 training / test split. In the zero-shot learning setting, a class-wise split was implemented to clearly distinguish the training and test class sets. In full-shot learning, a sample-wise split was performed within each class to maintain a balanced class distribution between the training and test sets with non-overlapping samples.

[0135] Adversarial Robustness Evaluation Setting To verify the effectiveness of RobustRetrieVAL in enhancing adversarial robustness and improving generalization performance, white-box feature space evasion attacks were generated using projected gradient descent (PGD) with various attack strengths on both real and synthetic data. For various attack strengths, we used various numbers of attack gradient steps (s∈{1, 10}) and

number

[0136] Results in data-deficient settings Domain Augmentation Figure 19 shows the results of applying DINO and ViT in the full-shot setting and data-scarcity settings, consistent with the above discussion, without and with RobustRetrieval (denoted by "RR-") to augment the data. That is, Figure 19 shows the results of automatic domain augmentation with RobustRetrieval. PD is the domain present in the original training data. AD is a superset of domains augmented based on class-specific characteristics. The domains (activity-based) for the Xcub data are flying (Af), sitting (As), and underwater (Aa). The domains (body-based) for the Xcars data are sedan (Bs), SUV crossover (Bc), sports performance, and convertible (Bp). Δ* represents the optimal diversity coefficient Δ used in ORDC filtering. In the full-shot setting, RobustRetrieval demonstrates significant improvements in image retrieval performance across multiple domain-scarcity scenarios across datasets and ViT models. As detailed in Figure 19, RobustRetrieval outperforms the baseline ViT model under all domain-scarcity conditions. Specifically, DINO H Models and ViT H The model increases the R@1 scores for the domain-specific Cars196Bp category by 18.3% and 20.1%, respectively. For the CUB-200-2011As domain, the increase is H 3.6% and DINO H The result is 2.6%. Therefore, the effectiveness of RobustRetrieVAL in enhancing representation in under-represented domains is demonstrated.

[0137] The performance of RobustRetrieVAL was rigorously evaluated in full-shot and zero-shot settings with varying degrees of class imbalance, parameterized by class imbalance relaxation κ. Figure 20 presents a table of results for applying DINO and ViT to class imbalance settings in full-shot and zero-shot learning. Specifically, Figure 20 shows the results of automatic class imbalance relaxation in zero-shot and full-shot learning tasks. Here, κ represents the number of training classes for λ = 2. The full-shot and zero-shot scenarios contain 200 and 100 training classes, respectively, for the CUB-200-2011 data, and 196 and 98 classes for the Cars196 data. A Δ of 1 was found to be optimal for the CUB-200-2011 data, and a Δ of 1.5 for Cars196. Figure 20 details the consistent performance of RobustRetrieVAL relative to the baseline Hyperbolic ViTs. In particular, RobustRetrieVAL achieved significant improvement in the zero-shot scenario with κ = 75, where there are very few training patterns. On the CUB-200-2011 dataset, the R@1 score is DINO H 11.7% than ViT H It increased by 5.4% compared to the previous year. H 18.8% than ViT H It was 18.1% stronger than the previous model.

[0138] Adversarial Robustness Gain The adversarial robustness improvement of RobustRetrieVAL in retrieval models was evaluated using customized experiments. Figure 21 shows the results of various l ∞ We show the R@1 gains against various strengths of white-box PGD attacks devised using noise bound ε and adversarial optimization gradient step s. These attacks optimize the target model (DINO) in the embedding space. H and RR-DINO H) to induce evasion. That is, plots (i) and (ii) on CUB-200-2011 and plots (iii) and (iv) on Cars196 data show the adversarial R@1 improvement achieved by the RobustRetrieVAL framework against white-box embedding spatial PGD attacks of various strengths. The robustness is measured using various numbers of attack gradient steps (s∈1,10) and

number

number

[0139] Standard Benchmark Results Figure 22 shows a table comparing the RobustRetrieveVAL (RR) trained model with standard performance benchmarks. H and RR-ViT H) outperforms the SotA model on the CUB-200-2011, Cars196, and SOP datasets, even when data augmentation is reduced with SOP due to class complexity and computational requirements. This demonstrates the effectiveness of RobustRetrieVAL in challenging data environments. The embedding size for all models was set to 128. Despite the fact that standard DML benchmark datasets typically have balanced class and domain distributions, RobustRetrieVAL is specifically designed to perform well in data-sparse scenarios. As shown in Figure 22, it outperforms the current SotA model (even with limited augmentation) on the CUB-200-2011, Cars196, and SOP datasets. We also observe that the ViT model initialization (ViT-S for CUB-200-2011 and DINO for Cars196) significantly benefits from pre-training. When pre-trained models have unique dataset-specific knowledge, it becomes difficult to enhance SotA without changing the model architecture or optimization strategy.

[0140] The above thorough evaluation demonstrates the role of RobustRetrieVAL in improving the generalizability of CBIR models, improving performance on both clean and adversarial samples. RobustRetrieVAL's targeted augmentation is particularly evident in limited-data scenarios, mitigating the scarcity of available data. The effectiveness of RobustRetrieVAL is also evident on evenly balanced CBIR benchmarks.

[0141] The image processing ML models referred to in the above descriptions of Figures 3-9 can be used for tasks including image retrieval after training. Applications of image retrieval include, for example, image retrieval of retail products, hazard detection systems, facial recognition, person re-identification, image search engines, and medical vision. The training methods disclosed herein are particularly useful for training image retrieval models for use in data-scarce scenarios, such as: Retail Security and Lost Item Search: A customer or staff member can use an image of a lost item, such as a wallet or shopping bag, to search the store's surveillance system to determine where the item was last seen or placed. That is, they can search frames of the video stream (from the surveillance system) for images that resemble that of the item / object. This is of course not limited to the retail / lost item setting, and other applications such as security, person identification, etc. are also considered. · Disaster response analysis: Identifying the extent of damage in post-disaster scenarios. Law enforcement vehicle search: If a particular vehicle is involved in a public safety incident, a trained image search model can search traffic camera feeds or public claims to find similar vehicles (similar to the retail / lost and found scenario above).

[0142] A computer-implemented framework / method for training image retrieval models through controlled augmentation of synthetic training data while automatically identifying missing training data and training weaknesses is disclosed. The framework includes: Receive a set of original training images and associated data labels and metadata; Generate image descriptions of the original training images using visual question answering and image captioning models; Analyze the generated image descriptions and available metadata, if any, using a hybrid approach that integrates heuristics and large-scale language models (LLMs) with prompt engineering to generate data summaries, class distribution insights, domain distribution insights, and identification of additional training classes; Selecting an Augmentation Protocol using an Augmentation Protocol Selector (APS), which develops reinforcement goals and generates reference instructions for creating text prompts, and which incorporates feedback from a Data Insight Generator, Filtering Feedback, and Training Feedback; · Create text prompts for text-picture models using LLM based on reinforcement objectives and reference descriptions from APS; Use a text-image model to generate synthetic images from text prompts, Compute class-specific centers in the embedding space using the average cluster radius and adjust the synthetic image using an Outlier Removal and Diversity Control (ORDC) method with an adjustable diversity coefficient. Ensure that the adjusted synthetic image meets the overall enhancement objectives and detect class or domain imbalances; Augmenting the original training images with filtered and modified synthetic images; · Training a deep metric learning model (image retrieval model) using augmented training data to improve image retrieval performance and adversarial robustness on clean data; Generate training feedback by evaluating model performance and identifying underperforming classes; Based on the training feedback, update the enhancement objectives and reference descriptions for underperforming classes and generate new synthetic data to merge with the training dataset for subsequent model training cycles.

[0143] In some implementations, the Data Insight Generator includes: a) an image description generation function that processes an image dataset and generates a description text using an image-text model or a visual question-answering model; b) a tokenization function that uses natural language processing techniques to split the generated image descriptions into word tokens; c) a stop word removal function that removes unnecessary tokens from the tokenized data to refine the dataset for further analysis; d) Part-of-speech tagging, which annotates cleaned tokens with grammatical information to extract meaningful insights; e) A frequency analysis function that identifies salient nouns and verbs from part-of-speech tagged tokens, providing a basis for contextual and domain-specific insights; f) Metadata extraction functionality that gathers contextual data from dataset metadata, which can be further refined using large-scale language models; g) Domain-specific insight inference capabilities that utilize frequency analysis results and refined metadata in combination with prompt engineering techniques to infer domain distribution insights; h) A class distribution assessment function that statistically assesses the balance of classes in the dataset and guides the augmentation process; i) a novel class discriminator that uses a large-scale language model to detect additional training classes not represented in the current dataset; j) a data summary synthesis function that combines domain insights, class distribution information, and novel class identification to develop a comprehensive data summary; k) Formatting capabilities to prepare the synthesized data summary for downstream use and ensure compatibility with augmentation strategies and model training processes.

[0144] In some implementations, the APS includes: a) an input interface that receives data insights including a data summary, class distribution information, absent classes, and domain imbalance information from the data insight generation unit of claim 2, together with filtering feedback and training feedback; b) an output interface that provides extended purpose and reference instructions for generating text prompts; c) A GenerateAugmentationObjective function that generates an initial per-class reinforcement prompt generation objective for each class in the class distribution using a large-scale language model (LLM); d) A GenerateReferenceDescription function that generates detailed reference descriptions for each class, providing the necessary context for downstream LLMs to perform text prompt generation; e) an IntegrateFeedback function that adjusts the reinforcement objective by integrating the initial objective with feedback received from the filtering feedback and training feedback, using the LLM to adapt the reinforcement strategy; f) A feedback integration mechanism that uses only the output from the Data Insights Generator during the initial training cycle to formulate reinforcement objectives, and then dynamically incorporates feedback in subsequent cycles to update the objectives and reference descriptions.

[0145] In some implementations, the LLM is configured to follow the instructions and information of the APS, generating text prompts specifically tailored to produce synthetic images that address the class and domain imbalances identified in the training data, as well as introducing newly identified additional training classes.

[0146] In some implementations, the ORDC method includes: a) A pre-trained deep metric learning model that maps the original and synthetic data into a feature-rich embedding space, enabling content-aware comparison beyond pixel-level differences; b) A class-specific centroid calculation mechanism that determines the centroid of each class in the original training data to contextualize outlier detection; c) an average distance metric calculation that evaluates the average embedding-space intra-class expansion (radius of embedding-space class clusters) within the original training data, which serves as the basis for defining outlier and diversity criteria; d) Introducing adjustable scaling to adaptively set outlier and diversity thresholds, and diversity coefficients that ensure the retention of synthetic samples that positively contribute to model training; e) an iterative process of dynamically adjusting the diversity factor to accommodate different levels of data complexity and diversity of application domains; f) A synthetic data filtering algorithm that selectively includes synthetic samples in the cleaned dataset based on their distance to class centroids, modulated by diversity counts according to (a)–(e).

[0147] In some implementations, the filter feedback component is configured to instruct the APS to generate additional synthetic data in response to detected imbalances caused by the ORDC cleaning process.

[0148] In some implementations, the Filtering Feedback Component uses an Imbalance Detection algorithm to ensure compliance with the Enhancement Objective and instructs the APS to generate compensation data when an imbalance is detected.

[0149] In some implementations, the training feedback generation component is configured to use the Recall@K metric to evaluate the clean data performance and adversarial robustness assessment of the trained model and to signal the APS for additional data generation for classes with low Recall@K scores on both clean and adversarial inputs.

[0150] In some implementations, the training feedback generation component employs a performance evaluation algorithm that evaluates model performance metrics to identify the highest-scoring models and signal the APS for target data generation for poorly performing classes.

[0151] In some implementations, APS updates its augmentation strategy based on training feedback to generate new synthetic data, which is merged with the existing training dataset to address specific weaknesses identified in model performance.

[0152] A system for generating augmented training data for image retrieval is disclosed, the system comprising: a processing unit configured to execute instructions; a memory unit storing instructions for performing the above-defined methods; and an interface configured to receive input data and provide output data, the system being configured to implement the above-defined hybrid data insight generation approach, APS, LLM, text-image model, ORDC method, and training feedback generation.

[0153] A non-transitory computer-readable medium is disclosed that stores instructions that, when executed by a computer, cause the computer to perform the above-described method.

[0154] In some implementations, a pre-trained deep metric learning model is used to extract the embeddings.

[0155] The methods and systems disclosed herein ensure continuous adaptation and refinement, leading to robust and accurate image retrieval models.

[0156] In general, problems associated with image retrieval models (and image processing models in general) include low accuracy and high adversarial susceptibility, which are caused (at least in part) by limited generalizability. Limited generalizability arises from the following causes: Lack of diverse data (lack of or missing training information), Poorly defined hypothesis space (poor model architecture), Poor training schema.

[0157] The methods and systems disclosed herein aim to solve the problems of various data shortages (insufficient or missing training information) and poor training schemas.

[0158] The disclosed method and system effectively leverages existing image-text and text-image models with LLM capabilities to efficiently and automatically generate useful synthetic data for training.

[0159] Limitations of existing solutions for training image processing ML models include the lack of the ability to target specific areas of data needing improvement and the need to fine-tune the model to generate synthetic data.

[0160] The methods and systems disclosed herein achieve the following advantages, among others: Enhanced generalization in trained image retrieval models improves clean data image retrieval performance and improves adversarial robustness against malicious attacks. Consistent Workflow: The data augmentation process is streamlined through a novel methodology that results in a novel single framework that effectively integrates image-text, LLM, and text-image models. · Automatic identification and augmentation of missing training information.

[0161] The high-quality synthetic data generated in a controlled manner by the disclosed methods and systems is useful for training highly accurate deep learning models even when real data is unavailable. Automated pipeline processing reduces the manual effort required for generating, preprocessing, and augmenting synthetic data. Leveraging a basic generative model for training image processing ML models results in highly accurate and safe image processing ML models.

[0162] The RobustRetrieval method is based on a probabilistic, approximately correct (PAC) learning framework. This framework evaluates learning algorithms based on the probability of selecting a nearly accurate hypothesis from a large number of training examples. The RobustRetrieval method effectively augments the training set by generating targeted synthetic data. The goal is to improve the PAC generalization bounds and, consequently, improve model accuracy with new, unseen data. The RobustRetrieval method leverages the expressive power of DNNs to exploit their ability to represent complex features and decision boundaries, which are essential for high-dimensional data in image retrieval tasks. Traditional data augmentation methods can unintentionally degrade performance due to untargeted augmentation of simple content or limited model capabilities. Unlike traditional methods, RobustRetrieval generates and augments the most effective synthetic data in fewer generations, reducing training costs. The ORDC module function ensures effective navigation of the optimization landscape by removing outliers and introducing controlled training complexity through diversity based on the optimization landscape theorem. For completeness, note that the optimization landscape theorem is a general DNN loss landscape optimization theorem, which provides various error bounds and performance guarantees regarding convergence of training to a global optimum during training on an objective function (i.e., the surface of the objective function (e.g., the shape and distribution of local minima, saddle points, and other critical points, and the paths connecting these points in the high-dimensional space in which neural network parameters reside)).

[0163] As mentioned above, empirical evaluation of RobustRetrieVAL highlights its effectiveness in enhancing DML model training. The observed performance gains reach 5.93% in data-sparse domains and 5.24% in class-sparse training data scenarios. Furthermore, this framework outperforms current state-of-the-art (SotA) vision transformer models, achieving a 1% improvement on standard balanced image retrieval benchmarks and a 2.3% improvement in balanced data-sparse scenarios. RobustRetrieVAL also achieves 1.9% higher adversarial robustness, especially against imperceptible adversarial attacks. These results highlight the contribution of RobustRetrieVAL to training more robust and generalizable models.

[0164] RobustRetrieVAL can be thought of as an integrated framework that automatically identifies and augments context-dependent, potentially missing training information while tracking training weaknesses. This enables the efficient generation and augmentation of relevant training data, patching of weak classes during training, improving performance on clean data, and improving the adversarial robustness of image retrieval models.

[0165] The methods and systems disclosed herein include / achieve, among other things: A novel integrated multimodal framework: a pioneering framework that integrates image-text, LLM, and text-image models for controlled synthetic data augmentation. Multiple advancements: Introduction of new custom components: Hybrid Data Insight Generator, Dynamic Augmentation Protocol Selector, Outlier Removal and Filtering, Filtering Feedback, Training Feedback to enable controlled data augmentation. Data Quality and Model Consistency: Addressing key issues in data quality and model consistency by compensating with high-quality synthetic data. Scalability and Adaptability: The proposed framework can be applied to process diverse datasets, enabling customization for specific AI applications. High impact in data-scarce applications: for example, highlighting the potential for interdisciplinary impact, particularly in areas where data scarcity and imbalances are a challenge. Contributing to AI security: Contributing to the security and integrity of trained image search models by improving their performance on clean data and robustness against adversarial attacks. Research Implications: Sets a new benchmark for synthetic data augmentation research, with implications for future research in data imbalance, domain adaptation, and continuous learning.

[0166] 23 is a block diagram of an information processing appliance 10 or computing device 10, such as a data storage server, that may be used to embody the present invention, to perform some or all of the operations of methods embodying the present invention, and to perform some or all of the tasks of the apparatus of the embodiments. The computing device 10 may be used to implement any of the method steps described above, such as any of steps S31-S36 and / or S51-S64, and / or any of the operations of the modules disclosed herein, such as any of the image-to-text model 31, DIG 32, APS 33, prompt generator 34, text-to-image model 35, ORDC module 36, and training feedback model 37, and / or any of the trained image processing ML models.

[0167] Computing device 10 includes a processor 993 and memory 994. Optionally, the computing device also includes a network interface 997 for communicating with other such computing devices, such as other computing devices of embodiments of the present invention. Optionally, the computing device also includes one or more input mechanisms, such as a keyboard and mouse 996, and a display unit, such as one or more monitors 995. These elements may facilitate user operation. The components may be connected to each other via a bus 992.

[0168] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause it to perform one or more functions or operations. For example, the computer-executable instructions may include instructions for implementing the methods disclosed herein or any of the method steps described above, e.g., any of steps S31-S36 and / or S51-S64, and / or any of the operation of the modules disclosed herein, e.g., any of the image-to-text model 31, DIG 32, APS 33, prompt generator 34, text-to-image model 35, ORDC module 36, and training feedback model 37, and / or any of the trained image processing ML models. The term "computer-readable storage medium" may include any medium capable of storing, encoding, or retaining a set of instructions for execution by a machine, causing the machine to perform any one or more of the method steps of the present disclosure; thus, the term "computer-readable storage medium" includes, but is not limited to, solid-state memory, optical media, and magnetic media. For example, without limitation, the term "computer-readable storage medium" may include solid-state memory, optical media, and magnetic media. By way of example, and without limitation, such computer-readable media may include non-transitory computer-readable media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM), or other optical disk storage, magnetic disk storage, or other magnetic storage, flash memory devices (e.g., solid-state memory devices), and the like.

[0169] The processor 993 is configured to control the computing device and perform processing operations, for example, to execute a computer program stored in the memory 994 to perform any of the method steps described above, for example, any of steps S31-S36 and / or S51-S64, and / or any of the operations of the modules disclosed herein, for example, any of the image-text model 31, DIG 32, APS 33, prompt generator 34, text-image model 35, ORDC module 36, and training feedback model 37, and / or any of the trained image processing ML models. The memory 994 stores data that is read and written by the processor 993, and may store original training data and / or synthetic training data and / or metadata and / or label information and / or LLM information and / or weights of image processing ML models and / or weights of other ML models and / or text data of LLM models and / or instructions and / or responses from LLM models and / or thresholds and / or performance metric values ​​and / or test data and / or feedback information and / or algorithms and / or input data and / or other data, and / or programs for performing any of the method steps or operations described above.

[0170] As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. A processor may also include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing other instruction sets or processors implementing a combination of instruction sets. A processor may also include one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. In one or more embodiments, a processor is configured to execute instructions to perform the steps and operations discussed herein. The processor 993 may be considered to include any of the modules described above. Any operation described as being performed by a module may be implemented as a method by a computer, e.g., the processor 993.

[0171] The display unit 995 may display representations of data stored by the computing device, such as original training data and / or synthetic training data and / or metadata and / or label information and / or LLM information and / or weights of image processing ML models and / or weights of other ML models and / or text data and / or instructions for LLM models and / or responses from LLM models and / or thresholds and / or performance metric values ​​and / or test data and / or feedback information and / or algorithms and / or input data and / or other data and / or GUI windows and / or interactive representations that allow a user to interact with the device 10, for example, by drag-and-drop or selection interactions, and / or any other output described above, and may also display a cursor and dialog boxes and screens that allow a user to interact with programs and data stored on the computing device. The input mechanism 996 may enable a user to input data and instructions into the computing device, such as allowing a user to enter any of the user inputs described above.

[0172] The network interface (network I / F) 997 can be connected to a network such as the Internet, and can be connected to other such computing devices via the network. The network I / F 997 can control input and output data to and from other devices via the network.

[0173] Other peripheral devices may be included in the computing device, such as a microphone, speakers, printer, power supply unit, fan, case, scanner, tracker ball, etc.

[0174] A method embodying the present invention may be performed on a computing device / appliance 10 such as that shown in FIG. 23. Such a computing device need not have all of the components shown in FIG. 23, but may be comprised of a subset of these components. For example, the device 10 may include a processor 993 and a memory 994 coupled to the processor 993. Alternatively, the device 10 may include a processor 993, a memory 994 coupled to the processor 993, and a display 995. A method embodying the present invention may be performed by a single computing device in communication with one or more data storage servers over a network. The computing device may also be a data storage device that stores at least a portion of the data.

[0175] Methods embodying the present invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the data.

[0176] The invention can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention can also be implemented as a computer program or computer program product, i.e., a computer program embodied in a non-transitory information carrier, for example a machine-readable storage device or a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0177] A computer program may be in the form of a stand-alone program, a computer program portion, or multiple computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, such as as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0178] The method steps or module operations of the present invention (e.g., any of steps S31-S36 and / or S51-S64, and / or the operations of any of the modules disclosed herein, e.g., image-text model 31, DIG 32, APS 33, prompt generator 34, text-image model 35, ORDC module 36, and any of the training feedback model 37, and / or the trained image processing ML model) can be performed by one or more programmable processors executing computer programs that perform the functions of the present invention by manipulating input data and generating output. The apparatus of the present invention can be implemented as programmed hardware or as dedicated logic circuitry, including field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc.

[0179] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memories that store instructions and data.

[0180] The above-described embodiments of the invention may be advantageously used independently of any other embodiment or in any feasible combination with one or more other embodiments.

[0181] The present disclosure extends to the following statements:

[0182] (S1) 1. A computer-implemented method comprising: generating image descriptions for the images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; 20. A computer-implemented method comprising:

[0183] (S2) determining the at least one domain and / or class that is unrepresented or underrepresented in the original training set, determining common terms within the image description; using a first LLM to infer, based on the general terminology, the domains represented in the original training set and the number of images in the original training set that represent each domain; The computer-implemented method of S1, comprising:

[0184] (S3) determining at least one domain and / or class that is unrepresented or underrepresented in the original training set, determining that a domain represented in the original training set is under-represented if the number or percentage of images in the original training set that represent that domain is below a class threshold; and / or using a fourth LLM to determine whether there is at least one domain that is not represented by any image in the original training set, and if it is determined that there is at least one domain that is not represented by any image in the original training set, determining the at least one domain as at least one unrepresented domain; The computer-implemented method of S1 or S2, comprising:

[0185] (S4) 10. The computer-implemented method of claim 1, wherein using the first LLM to infer the domains represented in the original training set and the number of images in the original training set that represent each domain includes using metadata of the original training set.

[0186] (S5) The computer-implemented method of S5, further comprising extracting metadata from / associated with the original training set and, using a fifth LLM, refining the extracted metadata as necessary for use in inferring the domains represented in the original training set and the number of images in the original training set that represent each domain.

[0187] (S6) The computer-implemented method of S5, wherein the metadata includes labels and / or label information associated with images in the original training set.

[0188] (S7) The computer-implemented method of S1, wherein determining at least one domain and / or class that is unrepresented or underrepresented in the original training set comprises determining the number of images in the original training set that are associated with / represent each class based on metadata and / or labels associated with the images.

[0189] (S8) determining at least one domain and / or class that is unrepresented or underrepresented in the original training set, determining that a class represented in the original training set is under-represented if the number or percentage of images in the original training set representing that class is below a class threshold; and / or using a sixth LLM to determine whether there is at least one class that is not represented by any images in the original training set, and if it is determined that there is at least one class that is not represented by any images in the original training set, determining the at least one class as at least one unrepresented class; The computer-implemented method of S1, comprising:

[0190] (S9) The computer-implemented method of S8, wherein determining the number of images in the original training set associated with / representing each class includes using a seventh LLM.

[0191] (S10) The computer-implemented method of S2, wherein determining general terms in the image descriptions includes determining the general terms using natural language processing (NLP) techniques.

[0192] (S11) determining common terms in the image description; tokenizing the image descriptions; removing stop words from the tokenized image description to provide cleaned image description tokens; assigning part-of-speech, POS tags to the cleaned image description tokens; extracting nouns and verbs from the cleaned image description tokens based on the POS tags; determining the n most frequently occurring nouns and verbs as said general terms; The computer-implemented method of S2, comprising:

[0193] (S12) The step of generating at least one instruction comprises: generating instructions naming the under-represented or under-represented classes when the under-represented or under-represented classes are determined; generating instructions naming the under-represented or unrepresented domain when the under-represented or unrepresented domain is determined; The computer-implemented method according to any one of S1 to 11, comprising:

[0194] (S13) 13. The computer-implemented method of any of S1-12, wherein generating at least one instruction comprises generating at least one instruction for each class represented in the original training set.

[0195] (S14) The step of generating at least one instruction comprises: 12. The computer-implemented method of any of S1-11, comprising, when under-represented or unrepresented domains have been determined, generating instructions for each class represented in the original training set, each instruction naming the under-represented or unrepresented domain.

[0196] (S15) A computer-implemented method described in any of S1 to 14, wherein the step of generating at least one instruction includes generating an enrichment objective including context for the third LLM for use in generating the at least one text prompt, and generating at least one reference description, each reference description naming a class and / or domain.

[0197] (S16) 15. The computer-implemented method of any of S1-14, wherein generating the at least one instruction includes generating an enrichment objective including a context for the third LLM for use in generating the at least one text prompt, and generating a reference description for each class represented in the original training set, each reference description naming an under-represented or under-represented domain and / or specifying whether the class is an under-represented class.

[0198] (S17) 15. The computer-implemented method of any of S1 to S14, wherein generating the at least one instruction includes generating an enrichment objective including a context for the third LLM for use in generating the at least one text prompt, and generating a reference description for each unexpressed class, each reference description naming the unexpressed class.

[0199] (S18) The computer-implemented method of any of S1 to 17, wherein the at least one instruction and / or the reinforcement objective and / or the at least one reference description specifies a number of text prompts to be generated.

[0200] (S19) 19. The computer-implemented method of any of S1-18, wherein the step of generating at least one text prompt includes the step of instructing the third LLM to generate a plurality of text prompts corresponding to each reference description and to use a reinforcement objective as a context for generating the text prompts.

[0201] (S20) The computer-implemented method of any of S1 to 19, wherein generating at least one text prompt includes generating a plurality of text prompts corresponding to each instruction.

[0202] (S21) A computer-implemented method described in any of S1 to 19, wherein the step of generating at least one text prompt includes the step of generating a specified number of text prompts for each instruction / each reference explanation / each reinforcement purpose.

[0203] (S22) A computer-implemented method described in any of S1 to 21, wherein generating the at least one text prompt includes generating the at least one text prompt such that the at least one text prompt includes a class and / or domain named in the at least one instruction.

[0204] (S23) 23. The computer-implemented method of any of S1-22, wherein generating the at least one composite image includes providing the at least one text prompt as input to the text-image model to generate a composite image for each of the at least one text prompt.

[0205] (S24) generating at least one composite image includes generating a composite set including a plurality of composite images; The computer-implemented method comprises: performing a cleaning process, the cleaning process including cleaning the composite set by removing any composite images determined to be outliers to generate a cleaned composite set of composite images; The computer-implemented method of any of S1-23, wherein the enhanced training set includes the original training set of images and the cleaned synthetic set of synthetic images.

[0206] (S25) The step of cleaning the composite set to generate a cleaned composite set includes: generating a first embedding of the images in the original training set; generating a second embedding of the synthetic images in the synthetic set associated with the classes represented in the original training set; Calculating the mean embedding of each class of images in the original training set (based on the labels / label information); For each class of images in the original training set, calculate the average distance of the distances of the first embeddings of the images of the class from the mean embedding of the class; For a second embedding, comparing a distance between the second embedding and a mean embedding of the corresponding class to a class outlier threshold based on the mean distance of the corresponding class, and removing the synthetic image corresponding to the second embedding from the synthetic set if the distance is greater than the class outlier threshold; The computer-implemented method of S24, comprising:

[0207] (S26) The computer-implemented method of S25, wherein the class outlier threshold for a given class comprises the average distance of the class multiplied by a diversity factor.

[0208] (S27) The computer-implemented method of S25 or S26, wherein generating the first and second embeddings includes using a (trained) deep metric learning model.

[0209] (S28) The computer-implemented method of any of S1 to 27, wherein the image processing ML model is configured to use the (trained) deep metric learning model.

[0210] (S29) The computer-implemented method of S25 or S26, wherein generating the first and second embeddings includes using a deep metric learning model used (trained) by the image processing ML model.

[0211] (S30) The computer-implemented method of S25 or S26, wherein generating the first and second embeddings comprises extracting a representation from an image of interest, the representation being the embedding.

[0212] (S31) The computer-implemented method of any of S25 to S30, wherein calculating a mean embedding for each class comprises calculating an element-wise (arithmetic) mean of the first embeddings corresponding to the class.

[0213] (S32) The computer-implemented method of any of S25 to S31, wherein the first (and second) embedding is a (one-dimensional) vector.

[0214] (S33) The computer-implemented method of any of S25 to S32, wherein the step of calculating the average distance of a given class includes the steps of calculating the distance of the first embedding of an image of the class from the mean embedding of the class, and calculating the average (mean) of the calculated distances.

[0215] (S34) The computer-implemented method of S33, wherein each distance comprises a Euclidean distance, a cosine distance, or a dot product distance.

[0216] (S35) The computer-implemented method includes the steps of performing a checking process; the checking process includes checking the cleaned composite set to determine whether additional composite images are needed, and if it is determined that additional composite images are needed, performing a cleaning compensation process; The cleaning compensation process comprises: generating, using the second LLM, at least one additional instruction for the third LLM to generate at least one text prompt; generating the at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model, wherein the enhanced training set of images includes the at least one additional composite image; The computer-implemented method of any one of S25 to S34, comprising:

[0217] (S36) The computer-implemented method of S35, wherein generating the at least one additional instruction includes generating an enrichment objective including a context for the third LLM for use in generating the at least one text prompt, and generating at least one reference description, each reference description naming a class and / or domain.

[0218] (S37) The computer-implemented method of S36, wherein the at least one additional instruction and / or the enhancement purpose and / or the at least one reference description specify a (new) number of several text prompts to be generated.

[0219] (S38) The computer-implemented method of S37, wherein generating at least one text prompt includes instructing the third LLM to generate a plurality of text prompts corresponding to each reference description and to use an enrichment objective as a context for generating the text prompts.

[0220] (S39) The computer-implemented method of S37, wherein generating at least one text prompt includes generating a plurality of text prompts corresponding to each additional instruction.

[0221] (S40) The computer-implemented method of S37, wherein generating at least one text prompt includes generating a specified number of text prompts for each additional instruction / each reference explanation / each reinforcement purpose.

[0222] (S41) The computer-implemented method of S37, wherein generating the at least one text prompt includes generating the at least one text prompt such that the at least one text prompt includes a class and / or domain named in the at least one additional instruction.

[0223] (S42) The computer-implemented method of S37, wherein generating the at least one composite image includes providing the at least one text prompt as input to the text-image model to generate additional composite images for each of the at least one text prompt.

[0224] (S43) The computer-implemented method of any of S35 to S42, further comprising continuously repeating the cleaning, checking, and cleaning compensation process until the checking process determines that no additional composite images are required or until a check threshold number of iterations have been performed, and wherein the enhanced training set includes the original training set of images and the cleaned composite set of composite images produced at each iteration of the cleaning process.

[0225] (S44) Checking the cleaned composite set to determine if additional composite images are needed includes: comparing the number of synthetic images in the cleaned synthetic set associated with each class and / or each domain with the number of instructions, references, or reinforcement text prompts corresponding to that class or domain; determining, for each class and / or each domain, that additional synthetic images are needed if the number of synthetic images is determined to be less than the number of text prompts; The computer-implemented method of any one of S35 to S43, comprising:

[0226] (S45) The computer-implemented method of any of S35 to S44, wherein generating at least one additional instruction includes generating at least one instruction corresponding to each class for which an additional composite image is determined to be required and / or corresponding to each domain for which an additional composite image is determined to be required.

[0227] (S46) The computer-implemented method of S45, wherein at least one instruction corresponding to a predetermined class and / or corresponding to a predetermined domain specifies a new number of text prompts according to a comparison between the number of synthetic images associated with the predetermined class or domain and the number of (previous) text prompts in a previous iteration.

[0228] (S47) The computer-implemented method includes performing a training feedback process; The training feedback process includes evaluating the performance of the trained image processing model to determine whether additional synthetic images are needed, and performing a weak class compensation process if it is determined that additional synthetic images are needed; The weak class compensation process includes: using the second LLM to generate at least one (further) additional instruction for the third LLM to generate at least one text prompt; generating, using the third LLM, the at least one text prompt for the text-image model based on the at least one (further) additional instruction; generating at least one (further) additional synthetic image based on the at least one text prompt using the text-image model, wherein the enhanced training set of images includes the at least one (further) additional synthetic image; The computer-implemented method of any one of S1 to S46, comprising:

[0229] (S48) The training feedback process includes: training the image processing ML model using the enriched training set of images to generate the trained image processing ML model; evaluating the performance of the trained image processing ML model using (multiple classes of) test images, and if the performance of the trained image processing ML model is below a performance threshold for any class of the test images, determining that additional synthetic images are required and determining that class as at least one weak class; The computer-implemented method of S47, comprising:

[0230] (S49) The computer-implemented method of any of S47 to S48, further comprising: generating a plurality of enhanced training sets each corresponding to a plurality of diversity coefficients, each enhanced training set including images from the original training set and images from a synthetic set generated using a corresponding diversity coefficient; and wherein the training feedback process comprises: separately training the image processing ML models using the images from the enhanced training sets to generate a plurality of trained image processing ML models each corresponding to the images from the enhanced training sets; evaluating performance of the plurality of trained image processing ML models to determine a trained image processing ML model with a best performance; and using the trained image processing ML model and the corresponding enhanced training set images in determining whether further synthetic images are required.

[0231] (S50) The computer-implemented method of S49, wherein generating the at least one (further) additional instruction comprises generating the at least one (further) additional instruction based on the at least one weak class.

[0232] (S51) The computer-implemented method of S49, wherein generating the at least one (further) additional instruction comprises generating the at least one (further) additional instruction that names the at least one weak class.

[0233] (S52) The computer-implemented method of S49, wherein the step of generating at least one (further) additional instruction includes generating an enrichment objective including a context for the third LLM to use in generating the at least one text prompt, and generating at least one reference description, each reference description naming a class and / or domain.

[0234] (S53) The computer-implemented method of S49, wherein the at least one (further) additional instruction and / or the enhancement purpose and / or the at least one reference description specifies a (new) number of several text prompts to be generated.

[0235] (S54) The computer-implemented method of S49, wherein generating at least one text prompt includes instructing the third LLM to generate a plurality of text prompts corresponding to each reference description and to use an enrichment objective as a context for generating the text prompts.

[0236] (S55) The computer-implemented method of S49, wherein generating at least one text prompt includes generating a plurality of text prompts corresponding to each (further) additional instruction.

[0237] (S56) The computer-implemented method of S49, wherein the step of generating at least one text prompt includes the step of generating a specified number of text prompts for each (further) additional instruction / each reference explanation / the reinforcement purpose.

[0238] (S57) The computer-implemented method of S49, wherein generating the at least one text prompt includes generating the at least one text prompt such that the at least one text prompt includes a class and / or domain named in the at least one (further) additional instruction.

[0239] (S58) The computer-implemented method of S49, wherein generating at least one further additional composite image includes providing the at least one text prompt as input to the text-image model to generate a further additional composite image for each of the at least one text prompt.

[0240] (S59) The computer-implemented method of S47 includes a step of continuously repeating / repeat- ing the training feedback process and the weak class compensation process until it is determined in the training feedback process that no further additional synthetic images are required or until a training threshold number of iterations have been performed (the enhanced training set includes the original training set of images and further additional synthetic images generated in each iteration of the weak class compensation process).

[0241] (S60) The computer-implemented method of S47, wherein evaluating the performance includes calculating a recall@k metric.

[0242] (S61) The computer-implemented method of any of S1 to S60, further comprising training the image processing ML model using the enriched training set of images.

[0243] (S62) The computer-implemented method of S61, wherein the computer-implemented method includes using the image processing ML model after training.

[0244] (S63) The computer-implemented method of any of S1 to S62, wherein the image processing ML model includes an image retrieval model.

[0245] (S64) The computer-implemented method of S63, wherein the image retrieval model is for searching among video frames for at least one image that is (most) similar to a query image.

[0246] (S65) The computer-implemented method of S64, wherein the query image includes an object and the video frames include video frames from a surveillance video.

[0247] (S66) The computer-implemented method of S64, wherein the query image includes a vehicle and / or the video frames include video frames from a traffic camera video.

[0248] (S67) The computer-implemented method of S63, wherein the image retrieval model is for facial recognition.

[0249] (S68) A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: generating image descriptions for images in the original training set of images using the image-text model; determining, based on said image descriptions, at least one domain and / or class that is (unrepresented or) underrepresented in said original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class (unrepresented or underrepresented in the original training set); generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; a computer program comprising:

[0250] (S69) An information processing device comprising: a memory; and a processor connected to the memory, the processor: Use the image-to-text model to generate image descriptions for the images in the original training set of images, determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); using a second LLM to generate at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class (unrepresented or underrepresented in the original training set); generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; An information processing device configured as follows.

[0251] In addition to the above-described embodiments, the following supplementary notes are also disclosed. (Supplementary Note 1) A computer-implemented method comprising: generating image descriptions for the images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; 20. A computer-implemented method comprising: (Supplementary Note 2) The step of determining at least one domain and / or class that is unrepresented or underrepresented in the original training set comprises: determining common terms within the image description; using a first LLM to infer, based on the general terms, the domains represented in the original training set and the number of images in the original training set that represent each domain; Including, determining the at least one unrepresented or underrepresented domain and / or class; determining that a domain represented in the original training set is under-represented if the number or percentage of images in the original training set that represent that domain is below a domain threshold; and / or using a fourth LLM to determine whether there is at least one domain that is not represented by any image in the original training set, and if it is determined that there is at least one domain that is not represented by any image in the original training set, determining the at least one domain as at least one unrepresented domain; 2. The computer-implemented method of claim 1, further comprising: (Supplementary Note 3) The step of determining at least one domain and / or class that is unrepresented or underrepresented in the original training set comprises: determining the number of images in the original training set associated with each class based on metadata and / or labels associated with the images; determining the at least one unrepresented or underrepresented domain and / or class; determining that a class represented in the original training set is under-represented if the number or percentage of images in the original training set representing that class is below a class threshold; and / or using a sixth LLM to determine whether there is at least one class that is not represented by any images in the original training set, and if it is determined that there is at least one class that is not represented by any images in the original training set, determining the at least one class as at least one unrepresented class; 3. The computer-implemented method of claim 1 or 2, further comprising: (Supplementary Note 4) The step of generating at least one instruction includes: generating instructions naming the under-represented or under-represented classes when the under-represented or under-represented classes are determined; generating instructions naming the under-represented or unrepresented domain when the under-represented or unrepresented domain is determined; 3. The computer-implemented method of claim 1 or 2, comprising: (Supplementary Note 5) The step of generating at least one composite image includes a step of generating a composite set including a plurality of composite images; The computer-implemented method comprises: performing a cleaning process, the cleaning process including cleaning the composite set by removing any composite images determined to be outliers to generate a cleaned composite set of composite images; 3. The computer-implemented method of claim 1 or 2, wherein the enhanced training set includes the original training set of images and the cleaned synthetic set of synthetic images. (Supplementary Note 6) The step of cleaning the composite set to generate the cleaned composite set includes: generating a first embedding of the images in the original training set; generating a second embedding of the synthetic images in the synthetic set associated with classes represented in the original training set; Computing the mean embedding of each class of images in the original training set; For each class of images in the original training set, calculating the average distance of the distances of the first embeddings of the images of that class from the mean embedding of that class; for each second embedding, comparing the distance between the second embedding and the mean embedding of the corresponding class to a class outlier threshold based on the mean distance of the corresponding class, and removing the synthetic image corresponding to the second embedding from the synthetic set if the distance is greater than the class outlier threshold; 6. The computer-implemented method of claim 5, comprising: (Supplementary Note 7) The computer-implemented method of Supplementary Note 6, wherein the class outlier threshold for a given class comprises the average distance of the class multiplied by a diversity factor. (Supplementary Note 8) Further comprising the step of executing a check process; the checking process includes checking the cleaned composite set to determine whether additional composite images are needed, and if it is determined that additional composite images are needed, performing a cleaning compensation process; The cleaning compensation process comprises: generating, using the second LLM, at least one additional instruction for the third LLM to generate at least one text prompt; generating the at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model; 6. The computer-implemented method of claim 5, comprising: (Supplementary Note 9) Further comprising the step of performing a training feedback process; The training feedback process includes: evaluating the performance of the trained image processing ML model to determine whether additional synthetic images are needed; and performing a weak class compensation process if it is determined that additional synthetic images are needed; The weak class compensation process includes: generating, using the second LLM, at least one additional instruction for the third LLM to generate at least one text prompt; generating the at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model; 3. The computer-implemented method of claim 1 or 2, comprising: (Supplementary Note 10) The training feedback process includes: training the image processing ML model using the enriched training set of images to generate the trained image processing ML model; evaluating the performance of the trained image processing ML model using test images, and if the performance of the trained image processing ML model is below a performance threshold for any class of the test images, determining that additional synthetic images are required and determining that class as at least one weak class; 10. The computer-implemented method of claim 9, comprising: (Supplementary Note 11) The computer-implemented method of Supplementary Note 10, comprising continuously repeating the training feedback process and the weak class compensation process until it is determined that no additional synthetic images are required in the training feedback process or until a training threshold number of iterations have been performed. (Supplementary Note 12) The computer-implemented method of any one of Supplementary Notes 1 to 2, further comprising training the image processing ML model using the enriched training set of images. (Appendix 13) The computer-implemented method of Appendices 1 or 2, wherein the image processing ML model comprises an image retrieval model. (Supplementary Note 14) A computer program that, when executed on a computer, causes the computer to perform a method, the method comprising: generating image descriptions for images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; a computer program comprising: (Supplementary Note 15) An information processing device, comprising: a memory; and a processor connected to the memory, wherein the processor: Use the image-to-text model to generate image descriptions for the images in the original training set of images, determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images including the original training set of images and the at least one synthetic image; An information processing device configured as follows. [Explanation of symbols]

[0252] 31 Image-Text Model 32 Data Insights Generation Department 33 Augmented Protocol Selector (APS) 34 Prompt Generation Unit (LLM) 35 Text-Image Model 36 Outlier Removal and Diversity Control (ORDC) Module 37 Training Feedback Module 40 pre-trained models

Claims

1. 1. A computer-implemented method comprising: generating image descriptions for images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image; 20. A computer-implemented method comprising:

2. determining at least one domain and / or class that is unrepresented or underrepresented in the original training set, determining common terms within the image description; using a first LLM to infer, based on the general terms, the domains represented in the original training set and the number of images in the original training set that represent each domain; Including, determining the at least one unrepresented or underrepresented domain and / or class; determining that a domain represented in the original training set is under-represented if the number or percentage of images in the original training set that represent that domain is below a domain threshold; and / or using a fourth LLM to determine whether there is at least one domain that is not represented by any image in the original training set, and if it is determined that there is at least one domain that is not represented by any image in the original training set, determining the at least one domain as at least one unrepresented domain; The computer-implemented method of claim 1 further comprising:

3. determining at least one domain and / or class that is unrepresented or underrepresented in the original training set, determining the number of images in the original training set associated with each class based on metadata and / or labels associated with the images; determining the at least one unrepresented or underrepresented domain and / or class; determining that a class represented in the original training set is under-represented if the number or percentage of images in the original training set representing that class is below a class threshold; and / or using a sixth LLM to determine whether there is at least one class that is not represented by any images in the original training set, and if it is determined that there is at least one class that is not represented by any images in the original training set, determining the at least one class as the at least one unrepresented class; The computer-implemented method of claim 1 or 2, further comprising:

4. The step of generating at least one instruction comprises: generating instructions naming the under-represented or under-represented classes when the under-represented or under-represented classes are determined; generating instructions naming the under-represented or unrepresented domain when the under-represented or unrepresented domain is determined; 3. The computer-implemented method of claim 1, comprising:

5. generating at least one composite image includes generating a composite set including a plurality of composite images; The computer-implemented method comprises: performing a cleaning process, the cleaning process including cleaning the composite set by removing any composite images determined to be outliers to generate a cleaned composite set of composite images; The computer-implemented method of claim 1 or claim 2, wherein the enhanced training set includes the original training set of images and the cleaned synthetic set of synthetic images.

6. Cleaning the composite set to generate the cleaned composite set includes: generating a first embedding of the images in the original training set; generating a second embedding of the synthetic images in the synthetic set associated with classes represented in the original training set; Computing the mean embedding of each class of images in the original training set; for each class of images in the original training set, calculating the average distance of the distances of the first embeddings of the images of that class from the mean embedding of that class; for each second embedding, comparing the distance between the second embedding and the mean embedding of the corresponding class to a class outlier threshold based on the mean distance of the corresponding class, and removing the synthetic image corresponding to the second embedding from the synthetic set if the distance is greater than the class outlier threshold; The computer-implemented method of claim 5 , comprising:

7. The computer-implemented method of claim 6 , wherein the class outlier threshold for a given class comprises the average distance of the class multiplied by a diversity factor.

8. and performing a check process. the checking process includes checking the cleaned composite set to determine whether additional composite images are needed, and if it is determined that additional composite images are needed, performing a cleaning compensation process; The cleaning compensation process comprises: using the second LLM to generate at least one additional instruction for the third LLM to generate at least one text prompt; generating the at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model; The computer-implemented method of claim 5 , comprising:

9. performing a training feedback process; The training feedback process includes: evaluating the performance of the trained image processing ML model to determine whether additional synthetic images are needed; and performing a weak class compensation process if it is determined that additional synthetic images are needed; The weak class compensation process includes: using the second LLM to generate at least one additional instruction for the third LLM to generate at least one text prompt; generating the at least one text prompt for the text-image model based on the at least one additional instruction using the third LLM; generating at least one additional composite image based on the at least one text prompt using the text-image model; 3. The computer-implemented method of claim 1, comprising:

10. The training feedback process includes: training the image processing ML model using the enriched training set of images to generate the trained image processing ML model; evaluating the performance of the trained image processing ML model using test images, and if the performance of the trained image processing ML model is below a performance threshold for any class of the test images, determining that additional synthetic images are required and determining that class as at least one weak class; 10. The computer-implemented method of claim 9, comprising:

11. 11. The computer-implemented method of claim 10, further comprising continuously repeating the training feedback process and the weak class compensation process until it is determined that no additional synthetic images are needed in the training feedback process or until a training threshold number of iterations have been performed.

12. The computer-implemented method of claim 1 or 2, further comprising training the image processing ML model using the enriched training set of images.

13. The computer-implemented method of claim 1 or 2, wherein the image processing ML model comprises an image retrieval model.

14. A computer program which, when executed on a computer, causes the computer to carry out a method, the method comprising: generating image descriptions for images in the original training set of images using the image-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); generating, using a second LLM, at least one instruction for a third LLM to generate at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image; a computer program comprising:

15. An information processing device comprising: a memory; and a processor connected to the memory, the processor: generating image descriptions for images in the original training set of images using the image-to-text model; determining, based on the image descriptions, at least one domain and / or class that is unrepresented or underrepresented in the original training set using at least one large-scale language model (LLM); using a second LLM to generate at least one instruction to a third LLM for generating at least one text prompt based on the determination of the at least one domain and / or class; generating the at least one text prompt for a text-image model based on the at least one instruction using the third LLM; generating at least one synthetic image based on the at least one text prompt using the text-image model; generating an enhanced training set of images for use in training an image processing machine learning (ML) model, the enhanced training set of images comprising the original training set of images and the at least one synthetic image; An information processing device configured as follows.

Images