Method and device for generating calibration dataset considering learning domain of artificial neural network model and optimizing artificial neural network model using same

The method generates a calibration dataset by using text prompts in a pre-trained image generation model to optimize artificial neural network models, addressing the challenges of data structure and legal issues, resulting in effective model optimization and performance enhancement.

WO2024237371A1PCT designated stage expired Publication Date: 2024-11-21SAPEON KOREA INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/KR2023/006732
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-17
Filing Date
2023-05-18
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

The challenge lies in obtaining a calibration dataset for optimizing artificial neural network models, particularly for lightweighting and personalization, as existing methods require significant effort and often result in data that is not structured appropriately or is prone to legal issues, with zero-shot quantization generating data that resembles random noise rather than real-world data.

Method used

A method and device that generate a calibration dataset by creating text prompts based on a task-specific word pool and inputting them into a pre-trained image generation model, utilizing a visual language model pre-trained with hundreds of millions of image-sentence pairs to produce specialized images for model optimization, which can be used for Post Training Quantization (PTQ) or pruning, while avoiding legal issues related to data privacy and copyright.

Benefits of technology

This approach allows for the securement of a large amount of images optimized for the target model, enhancing model performance and usability for various purposes, including personalized optimization of distributed models, with improved accuracy and reduced effort, as demonstrated by experimental results showing comparable test accuracy to human-inspected datasets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure KR2023006732_21112024_PF_FP_ABST
    Figure KR2023006732_21112024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed are a method and a device for generating a calibration dataset considering a learning domain of an artificial neural network model and optimizing an artificial neural network model using same. According to one aspect of the present disclosure, a computer-implemented method is provided, the method comprising: a process for generating a text prompt on the basis of a task-specific word pool associated with a task of a pre-trained artificial neural network model; and a process for inputting the text prompt into a pre-trained image generation model so as to obtain a calibration dataset for optimizing the artificial neural network model.
Need to check novelty before this filing date? Find Prior Art

Description

Method and device for generating a calibration dataset considering the learning domain of an artificial neural network model and optimizing the artificial neural network model using the same

[0001] The present disclosure relates to a method and apparatus for generating a calibration dataset by considering the learning domain of an artificial neural network model and optimizing the artificial neural network model using the calibration dataset.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] As concerns about privacy in training and / or validation data intensify, it is becoming increasingly impossible to obtain data sets from model distribution entities for model miniaturization, personalization, or various other purposes.

[0004] To secure a dataset, data can be collected manually or automatically through web crawling. However, these methods ultimately require extensive effort, such as human verification of each and every sample, and securing a large amount of data is difficult compared to the effort required.

[0005] Recently, active research has been conducted on zero-shot quantization, a solution that utilizes self-generated data rather than securing a separate dataset. However, the data generated through this technique is closer to random noise than real-world data, like the data used in the actual training of the model. Therefore, users conducting lightweighting efforts face the challenge of verifying whether the data is appropriately structured for the target model.

[0006] The present disclosure aims to provide a method and device for obtaining an image set specialized for a target model to be optimized.

[0007] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] According to one aspect of the present disclosure, a computer-implemented method is provided, comprising: generating a text prompt based on a task-specific word pool associated with a task of a pre-trained target model; and inputting the text prompt into a pre-trained image generator model to obtain a calibration dataset for optimizing the target model.

[0009] According to another aspect of the present disclosure, a device for optimizing a pre-trained target model is provided, comprising: a text generation module for generating a text prompt based on a task-specific word pool associated with a task of the target model; and an image generation module for inputting the text prompt into a pre-trained image generation model to obtain a calibration dataset for optimizing the target model.

[0010] According to another aspect of the present disclosure, there is provided a computer-readable recording medium having stored thereon a command, which, when executed by a computer, causes the computer to perform the steps of: generating a text prompt based on a task-specific word pool associated with a task of a pre-trained target model; and inputting the text prompt into a pre-trained image generation model to obtain a calibration dataset for optimizing the target model.

[0011] According to embodiments of the present disclosure, a large number of images specialized for the target model's task can be secured by utilizing an image generation model pretrained on hundreds of millions of image-sentence pairs. The secured images can be used as a calibration set for model reduction, such as post-training quantization (PTQ) or pruning, or for various other applications.

[0012] According to embodiments of the present disclosure, images can be secured free from various legal issues, such as data privacy and copyright. These secured images can also be useful for personalized optimization of deployed models.

[0013] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0014] FIG. 1 is a block diagram schematically illustrating the configuration of an electronic device for generating and using a calibration data set according to one embodiment of the present disclosure.

[0015] FIG. 2 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure.

[0016] FIG. 3 is a flowchart illustrating the operation of a system using an electronic device according to one embodiment of the present disclosure.

[0017] FIG. 4 is a flowchart illustrating the operation of a system using an electronic device according to another embodiment of the present disclosure.

[0018] Figure 5 is a schematic diagram of a computing device to which the present disclosure can be applied.

[0019] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0020] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0021] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0022] FIG. 1 is a block diagram schematically illustrating the configuration of an electronic device for generating and using a calibration data set according to one embodiment of the present disclosure.

[0023] The electronic device (10) can generate a calibration dataset (150) for the target model (12) and / or optimize the target model based on the generated calibration dataset. The electronic device (10) can perform inference using the optimized model (14) and / or distribute the optimized model (14).

[0024] Here, the target model (12) may be a task-specific vision model that has been trained to perform a specific vision task. For example, the target model may be an artificial neural network model trained to perform a classification task, a segmentation task, and / or an object detection task on an input image.

[0025] The electronic device (10) can secure a calibration dataset by utilizing a pre-trained image generator of a visual language model with hundreds of millions of image and sentence pairs. Here, the visual language model can be configured to generate synthetic images in the visual recognition domain, which is a domain that ordinary people can commonly perceive. In the present disclosure, the term "image generator" may be used interchangeably with the term "image generation model."

[0026] To this end, an electronic device (10) according to an embodiment of the present disclosure may include all or part of a word pool configuration module (100), a text generation module (120), an image generation module (140), and an optimization module (160). Not all blocks illustrated in FIG. 1 are essential components, and some blocks included in the electronic device (10) in other embodiments may be added, changed, or deleted. Meanwhile, the components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in a form in which they are integrated with each other in an actual physical environment.

[0027] The word pool configuration module (100) can configure a task-specific word pool (110) based on information about the dataset domain learned by the target model (12).

[0028] If the target model (12) is trained to perform a task for a specific class set, the task-specific word pool (110) may include labels for each of the multiple classes constituting the class set. For example, if the target model (12) is trained to perform a task of classifying 1,000 classes, 1,000 words (e.g., 'dog', 'cat', etc.) corresponding to each class may be the primary elements constituting the task-specific word pool (110).

[0029] To this end, the word pool configuration module (100) can collect metadata including information about the dataset domain learned by the target model (12) from the server distributing the target model (12), but is not limited to this example.

[0030] The text generation module (120) can generate one or more text prompts to be input into the image generation model based on a task-specific word pool (110).

[0031] The text generation module (120) can generate one or more text prompts by applying a predefined rule to at least one word selected from a task-specific word pool (110). For example, the text generation module (120) can generate one or more text prompts in a rule-based manner borrowed from prompt engineering in visual-language models.

[0032] Rules for writing text prompts can be defined in advance, taking into account the tasks of the target model.

[0033] Table 1 shows examples of text prompt writing rules for models whose goal is classification tasks.

[0034] Rules for writing delimited text prompts(1)A(n) [IMAGE_TYPE] of a(n) [CLASS_LABEL](2)A(n) [high-quality / high-resolution / super realistic / detailed] [IMAGE_TYPE] of a(n) [CLASS_LABEL](3)A(n) [IMAGE_TYPE] of a(n) [ADJECTIVE] [CLASS_LABEL] [ADVERB](4)A(n) [IMAGE_TYPE] of a(n) [CLASS_LABEL]; [CLASS_DESC]

[0035] Referring to (1) of Table 1, the text prompt may include a word combination indicating an image type and a class label ([CLASS_LABEL]). The text generation module (120) may insert a class label ([CLASS_LABEL]) randomly selected from a task-specific word pool (110) into a predetermined position within the text prompt. The word combination indicating the image type may be composed of one or more words, such as, for example, "a [IMAGE_TYPE] of." Here, the image type ([IMAGE_TYPE]) may be randomly selected from a list of image types. The list of image types may include, but is not limited to, one or more words from among 'photo', 'illustration', 'snapshot', 'picture', 'drawing', and 'image'.

[0036] That is, the text generation module (120) can generate a text prompt in a format that describes an image containing an object corresponding to a class label. This is an extension of CLIP's prompt engineering to generate text prompts for image synthesis for calibration of a classification model. In the case of an image classification model trained by multi-modal learning such as CLIP, the classification performance is improved when the object included in the image is clearly indicated as being to be recognized, such as "a photo of a [CLASS_LABEL]", rather than when only a class label is presented. In addition, since most classification images have one object per image, it is expected that using a text prompt in the format of (1) of Table 1 will be helpful in generating classification images.

[0037] Referring to (2) to (4) of Table 1, the text generation module (120) may also generate a text prompt with additional explanation of the target to be generated.

[0038] First, referring to (2) of Table 1, the text prompt may further include one or more word combinations that modify the image type ([IMAGE_TYPE]). The word combinations that modify the image type may be randomly selected from a predefined list of image modifiers. The list of image modifiers may include, but is not limited to, one or more of 'high-quality', 'high-resolution', 'super realistic', and 'detailed', for example.

[0039] Referring to (3) of Table 1, the text prompt may further include a class label ([CLASS_LABEL]) or one or more word combinations ([ADJECTIVE] and / or [ADVERB]) that modify at least a portion of the preceding sentence. The word combinations are modifiers composed of one or more words, such as, but not limited to, adjectives, adverbs, adjective phrases, and / or adverbial phrases.

[0040] The text generation module (120) can extract word combinations to be added to a text prompt from a modifier pool consisting of one or more modifiers. For example, the text generation module (120) can extract one or more modifiers from a modifier pool that includes modifiers that can be compared (or contrasted) with each other (e.g., big - small, or faded - outstanding, etc.) and insert them into a predetermined position within the text prompt. As another example, the text generation module (120) can initially generate a text prompt using modifiers extracted from a modifier pool consisting of random words, and then correct the generated text prompt using a tool such as a language corrector.

[0041] Referring to (3) of Table 1, the text prompt may further include an additional description ([CLASS_DESC]) for the class label ([CLASS_LABEL]). The additional description ([CLASS_DESC]) may include a comment (or example) that briefly defines the class label ([CLASS_LABEL]). For example, if the class label ([CLASS_LABEL]) is 'hen', an additional description ([CLASS_DESC]) such as 'adult female kitchen' may be added. The text generation module (120) may search for the additional description ([CLASS_DESC]) for the class label ([CLASS_LABEL]) in a lexical database (e.g., WordNet), but is not limited to such examples.

[0042] Table 2 shows examples of text prompts generated according to the rules described in (d) of Table 1.

[0043] SeparatorText Prompta"a picture of a Vulpes macrotis; small gray fox of southwestern United States; may be a subspecies of Vulpes velox"b"a snapshot of a Siberian husky; breed of sled dog developed in northeastern Siberia; they resemble the larger Alaskan malamutes"c"a picture of an English springer spaniel; a breed having typically a black-and-white coat"d"an image of a gazelle; small swift graceful antelope of Africa and Asia having lustrous eyes"

[0044] Meanwhile, the method of obtaining a combination of words to decorate at least one sentence component within a text prompt is not limited to the above-described example, and various methods may be applied depending on the implementation example.

[0045] The text generation module (120) may also use modified text prompt writing rules as shown in Table 3 for data augmentation.

[0046] Rules for writing delimited text prompts (1)A(n) [IMAGE_TYPE] of a(n) [CLASS_LABEL_1] and a(an) [CLASS_LABEL_2](2)A(n) [high-quality / high-resolution / super realistic / detailed] [IMAGE_TYPE] of a(n) [CLASS_LABEL_1] and a(an) [CLASS_LABEL_2](3)A(n) [IMAGE_TYPE] of a(n) [ADJECTIVE_1] [CLASS_LABEL_1] and a(an) [ADJECTIVE_2] [CLASS_LABEL_2] [ADVERB]

[0047] Referring to (1) of Table 3, the text prompt may include multiple class labels ([[CLASS_LABEL_1]] and [CLASS_LABEL_2]) selected from a task-specific word pool. That is, the text generation module (120) may generate a text prompt in a format that describes an image including multiple objects corresponding to different class labels.

[0048] Referring to (2) and (3) of Table 3, the text prompt may further include one or more word combinations that modify the image type ([IMAGE_TYPE]) and / or one or more word combinations that modify each label ([ADJECTIVE_1] and [ADJECTIVE_2]). In addition, the text prompt may further include one or more word combinations that modify multiple labels simultaneously ([ADVERB]).

[0049] The rules described in Table 3 are a natural language implementation of the Mixup algorithm, which mixes two samples from an image, suggesting that data augmentation algorithms in the image domain can be directly implemented in rule-based text prompts.

[0050] Meanwhile, two or more of the rules exemplified in Table 1 or Table 3 may be combined and used. For example, the text generation module (120) may generate a text prompt based on a rule in the format of "A(n) [high-quality / high-resolution] [photo / illustration / snapshot] of a(n) [ADJECTIVE_1] [CLASS_LABEL_1] and a(n) [ADJECTIVE_1] [CLASS_LABEL_2] [ADVERB]", which is a combination of rules (2) and (3) of Table 3. Table 4 shows examples of text prompts generated according to the above-described rules.

[0051] Classification Text Prompta"a super realistic drawings of a large house finch and big a silky terrier in the forest"b"a high-quality drawings of a large peacock and a red wine in the water"c"a high-quality drawings of a big barbell and a large cheeseburger in the forest"d"a super realistic illustration of a schooner and a small black stork"

[0052] Meanwhile, if the task of the target model (12) is a segmentation task rather than a classification task, the text generation module (120) may generate a text prompt in a general descriptive sentence format rather than a format that describes an image. For example, the text generation module (120) may generate a text prompt that describes the actions or states of multiple objects corresponding to the same class. Here, the same class may be a class corresponding to a specific label randomly selected from a task-specific word pool (110).

[0053] These text prompt writing rules take into account that models for segmentation tasks are primarily trained using datasets that contain a large number of images containing multiple objects and a variety of surrounding environments other than the objects.

[0054] Table 5 shows examples of text prompt writing rules for models whose goal is the segmentation task.

[0055] Rules for Writing Quantifier Text Prompts (1) [Quantifiers] [CLASS_LABEL] [Predicate] (2) [Quantifiers] [CLASS_LABEL] [Predicate] [Supplemental phrase / clause]

[0056] The text prompt may include a plural representation of the class label. For example, the text prompt may further include a quantifier (e.g., Many) that modifies the class label ([CLASS_LABEL]). If the class label ([CLASS_LABEL]) is a countable noun, the text generation module (120) may reference a vocabulary database to obtain a plural noun for the corresponding class label ([CLASS_LABEL]) and insert the obtained plural noun into the position of the class label ([CLASS_LABEL]) within the text prompt, but is not limited to these examples.

[0057] To describe the actions or states of objects, the text prompt may further include a predicate and / or a supplementary phrase / clause. Here, the supplementary phrase / clause is a sentence element that describes a secondary situation related to the objects, and may include, for example, an adverbial phrase or adverbial clause. The supplementary phrase / clause may be added to the text prompt to increase the diversity of image generation.

[0058] Table 6 shows examples of text prompts generated according to the rules described in (b) of Table 5.

[0059] Distinct text prompts a "Many cars are running on the highway" b "Many cars are on the highway"

[0060] In the above, an example in which the text generation module (120) generates text prompt(s) based on predefined rules has been described, but the present disclosure is not limited thereto. In another example, the text generation module (120) may obtain text prompt(s) using a pre-trained language model based on a neural network. For example, the text generation module (120) may randomly select one or more words from a task-specific word pool (110) and input each selected word (or a combination of two or more words) into a language model to obtain one or more text prompts.

[0061] The image generation module (140) can input the generated text prompt into a pre-trained image generation model to obtain a calibration dataset (150) for optimizing the target model (12). Here, the image generation model may be known as DALL-E mini, DALL-E mega, DALL-E V1, DALL-E V2, Imagen, etc., but is not limited to these examples.

[0062] The image generation module (140) can input one or more text prompts into the image generation model, thereby obtaining a synthetic image corresponding to each text prompt. Inputting the text prompts into the image generation model may include transmitting the text prompts to a server executing the image generation model. The image generation module (140) can collect the synthetic image(s) output by the image generation model to form a calibration dataset (150).

[0063] The higher the completeness of the sentence (i.e., text prompt) input to the image generation model, the more suitable the image can be generated for the target model.

[0064] The optimization module (160) can optimize the target model (12) using the calibration dataset (150). For example, the optimization module (160) can adjust the quantization parameters based on the calibration dataset (150) during the process of performing post-training quantization (PTQ) on the target model (12). As another example, the optimization module (160) can determine the connection weights to be removed based on the calibration dataset (150) during the process of performing pruning on the target model (12). For this purpose, each synthetic image constituting the calibration dataset (150) may be labeled with a word (e.g., a class label) extracted from a task-specific word pool (110) to generate the corresponding synthetic image. As another example, the value output by the model before being lightweighted for each sample constituting the calibration dataset (150) may be used as a soft label.

[0065] Meanwhile, the above description has been made as an example of model optimization, namely, model weighting for a model for which training has been completed. However, the present disclosure is not limited thereto. That is, the calibration dataset (150) may be utilized in various applications. For example, the calibration dataset (150) may be applied to learning that takes into account passive quantization (QAT; Quantization Aware Training). As another example, the calibration dataset (150) may be used in cases where additional adversarial leaning is required in the process of implementing a model that can defend against backdoor attacks (a backdoor-attack defensive model).

[0066] Below, experimental results for calibration datasets generated according to various embodiments of the present disclosure are described with reference to Tables 7 to 9.

[0067] In this experimental example, to verify whether synthetic images can actually be used as a calibration dataset, the test accuracy of the quantization results of the artificial neural network was compared for each calibration dataset. Here, moving min / max was used as the calibration method for quantization.

[0068] In this experimental example, the artificial neural network to be quantized was a ResNet series neural network trained to perform classification tasks based on the ImageNet1K dataset, and two types of test datasets, ImageNetV0 and ImageNetV2, were used. Here, ImageNetV0 refers to the validation dataset of ImageNet1K used in ISLVRC2012, and ImageNetV2 is a test dataset constructed based on web crawling and other inspection tasks.

[0069] In this experimental example, text prompts written according to various text prompt writing rules were input into an image generation model to generate synthetic images. Hereinafter, the calibration datasets consisting of synthetic images generated based on each writing rule are referred to as DALLE-V0.1 to DALLE-V0.5. Table 7 shows the text prompt writing rules applied during the creation of each calibration dataset.

[0070] Calibration DatasetImage CreationModelText Prompt Writing RulesDALLE-V0.0DALL-E minion illustration of [CLASS_LABEL]DALLE-V0.1a photo of [CLASS_LABEL]DALLE-V0.2a [high-quality / super realistic] [illustration / drawing] of [CLASS_LABEL1] and [CLASS_LABEL2]DALLE-V0.3a [super realistic / high-resolution / detailed] [picture / image / photo / snapshot] of [CLASS_LABEL]; [CLASS_DESC]DALLE-V0.4DALL-E megaa [high-resolution / super realistic / high quality] [photo / image / picture] of [CLASS_LABEL] [ADV]DALLE-V0.5a [high-resolution / super realistic / high quality] [photo / image / picture] of [CLASS_LABEL] [ADV]; [CLASS_DESC]

[0071] As shown in Table 7, the known DALL-E mini model and DALL-E mega model were used as image generation models for generating synthetic images. Specifically, the DALL-E mini model was used to generate DALLE-V0.0 to DALLE-V0.3, and the DALL-E mega model, which has approximately twice as many parameters as the DALL-E mini model, was used to generate DALLE-V0.4 and DALLE-V0.5.

[0072] Tables 8 and 9 show the accuracy measurements for two test datasets, ImageNetV0 and ImageNetV2, after quantizing the ResNet family of artificial neural networks using various types of calibration datasets. Top-1 Accuracy and Top-5 Accuracy were used as accuracy metrics. In Tables 8 and 9, the two highest accuracy cases for each artificial neural network are highlighted in bold and underlined.

[0073]

[0074]

[0075] In Tables 8 and 9, 'FP32' is the result learned with the ImageNet training dataset before quantization, and 'uniform random' indicates the case where each pixel of the sample of the calibration dataset is sampled from a uniform distribution (x_{i}~U(-1,1)).

[0076] Comparing the results for DALLE-V0.0 and DALLE-V0.5, datasets built from synthetic images, with those for ImageNet V0 and ImageNet V2, datasets manually audited, reveals a wide range of accuracy outcomes, including slight decreases, similar accuracy, and even higher accuracy. This suggests that synthetic images can be used as a calibration dataset.

[0077] Furthermore, in DALLE-V0.3 and DALLE-V0.5, where text prompt writing rules are more complex or various decorative phrases are used, we can see that the test accuracy is improved overall.

[0078] FIG. 2 is a flowchart illustrating the operation of an electronic device according to one embodiment of the present disclosure. The method illustrated in FIG. 2 can be implemented by executing the functions of one or more components of the aforementioned electronic device (10) by at least one computing device. Therefore, the following description can also be understood in terms of operations performed by the computing device.

[0079] The electronic device (10) can generate a text prompt based on a task-specific word pool associated with a task of an artificial neural network model (S200). Here, the task-specific word pool can be configured based on a data set domain learned by the artificial neural network model. For example, if the artificial neural network model is trained to perform a classification task or a segmentation task for a predefined class set, the task-specific word pool can include labels of each of a plurality of classes constituting the predefined class set. Here, the classification task can mean a task of classifying the class of an object included in an input image based on the predefined class set. The segmentation task can mean a task of segmenting an input image into a plurality of regions based on the predefined class set.

[0080] In one embodiment, the electronic device (10) may obtain a text prompt by inputting at least one word selected from a task-specific word pool into a neural network-based pre-trained language model.

[0081] In another embodiment, the electronic device (10) may generate a text prompt by applying predefined rules to at least one word selected from a task-specific word pool. The electronic device (10) may apply different text prompt generation rules depending on the task of the artificial neural network model.

[0082] When the task of the artificial neural network model is a classification task, the electronic device (10) can randomly select one or more labels from a task-specific word pool, and generate a text prompt that describes an image including one or more objects, each corresponding to one or more labels, using the selected one or more labels. The text prompt may include, for example, a first word combination indicating a type of image, one or more labels selected from the task-specific word pool, and one or more second word combinations that modify each label. As another example, the text prompt may include a first word combination indicating a type of image, a plurality of labels selected from the task-specific word pool, and a second word combination that simultaneously modifies the plurality of labels. Here, each of the first word combination and the second word combination(s) may include one or more words.

[0083] When the task of the artificial neural network model is a segmentation task, the electronic device (10) can randomly select a label from a task-specific word pool and generate a text prompt describing the actions or states of multiple objects corresponding to the same class using the selected label. The text prompt can include, for example, a plural expression of the label selected from the task-specific word pool and a phrase or clause describing a contingencies associated with the selected label.

[0084] The electronic device (10) can input text prompts into a pre-trained image generation model to obtain a calibration dataset for optimizing an artificial neural network model (S220). For example, the electronic device (10) can input a plurality of text prompts obtained by performing process S200 multiple times into the image generation model, and collect synthetic images output by the image generation model in response to each text prompt to form a calibration dataset.

[0085] After step S220, the electronic device (10) may perform optimization on the artificial neural network model based on the acquired calibration dataset. Optimization on the artificial neural network model may include weight reduction of the artificial neural network model. For example, the electronic device (10) may perform post-training quantization (PTQ) based on the calibration dataset. As another example, the electronic device (10) may perform pruning based on the calibration dataset. After weight reduction of the artificial neural network model according to various weight reduction algorithms, the electronic device (10) may use the calibration data to correct the weight reduction model or various parameters for weight reduction.

[0086] After process S220, the electronic device (10) may transmit the acquired calibration dataset to another device.

[0087] FIG. 3 is a flowchart illustrating the operation of a system using an electronic device according to one embodiment of the present disclosure.

[0088] The first server (30) can distribute an artificial neural network model for which training has been completed (S300). The first server (30) may be a server that performs training on an artificial neural network model.

[0089] The electronic device (10) can generate text prompt(s) based on information about the artificial neural network model (S320). Process S320 may be, for example, identical to or corresponding to process S200 of FIG. 2.

[0090] The electronic device (10) can transmit the generated text prompt(s) to a second server (32) (S340). The second server (32) may be a server that executes inference using an image generation model. The electronic device (10) can transmit an image generation request including the text prompt(s) to the second server (32) via an API.

[0091] The electronic device (10) can obtain a composite image(s) generated based on text prompt(s) from the second server (32) (S350).

[0092] The electronic device (10) can build a calibration dataset using the acquired synthetic image(s) (S360).

[0093] The electronic device (10) can optimize an artificial neural network model using a calibration dataset (S380).

[0094] FIG. 4 is a flowchart illustrating the operation of a system using an electronic device according to another embodiment of the present disclosure. In describing FIG. 4, steps S400 and S420 to S460 may be identical to or corresponding to steps S300 to S360 of FIG. 3, and thus, detailed description of overlapping content will be omitted.

[0095] The first server (30) can distribute an artificial neural network model for which learning has been completed (S400).

[0096] The electronic device (10) can obtain information about the dataset domain learned by the artificial neural network model (hereinafter, "learning domain information") from the first server (30) (S410). For example, if the artificial neural network model is learned to perform a classification task and / or a segmentation task, the learning domain information may include the label names of each of a plurality of classes that the artificial neural network model can classify or segment.

[0097] The electronic device (10) can generate text prompt(s) based on the acquired learning domain information (S420).

[0098] The electronic device (10) can transmit the generated text prompt(s) to the second server (32), and in response, obtain the composite image(s) generated based on the text prompt(s) from the second server (32) (S430 and S440).

[0099] The electronic device (10) can build a calibration dataset using the acquired synthetic image(s) (S460). The electronic device (10) can transmit the acquired calibration dataset to the terminal (40) (S470).

[0100] The terminal (40) can optimize an artificial neural network model using a calibration dataset (S480). That is, according to another embodiment of the present disclosure, the entity optimizing the artificial neural network model may be the terminal (40), not the electronic device (10).

[0101] FIG. 5 is a block diagram schematically illustrating an exemplary computing device to which the present disclosure can be applied.

[0102] Referring to FIG. 5, the computing device (50) may include some or all of a memory (500), a processor (520), storage (540), an input / output interface (560), and a communication interface (580). The computing device (50) may structurally and / or functionally include at least a portion of the electronic device (10). The computing device (50) may be a stationary computing device such as a desktop computer, a server, and / or an intelligent camera, as well as a mobile computing device such as a smart phone and / or a laptop computer.

[0103] The memory (500) can store a program that causes the processor (520) to perform a method(s) according to one embodiment of the present invention. For example, the program can include a plurality of instructions executable by the processor (520), and the operation of the electronic device (10) described above can be performed by executing the plurality of instructions by the processor (520).

[0104] The memory (500) may be a single memory or multiple memories. Information required for the operation of the aforementioned electronic device (10) may be stored in a single memory or may be divided and stored in multiple memories. If the memory (500) is composed of multiple memories, the multiple memories may be physically separated. The memory (500) may include at least one of volatile memory and non-volatile memory. Volatile memory includes Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM), and non-volatile memory includes Flash memory.

[0105] The processor (520) may include at least one core capable of executing at least one instruction. The processor (520) may be a single processor or multiple processors. The processor (520) may execute instructions stored in the memory (500).

[0106] For example, the processor (520) can generate a text prompt based on a task-specific word pool associated with a task of the target model by executing instructions stored in the memory (500), and input the text prompt into a pre-trained image generation model to obtain a calibration dataset for optimizing the target model.

[0107] Storage (540) can maintain stored data even when power supplied to the computing device (50) is cut off. For example, storage (540) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk.

[0108] The storage (540) can store data to be processed by the processor (520) and data processed by the processor (520). For example, the storage (540) can store text prompt(s) generated by the processor (520) and / or synthetic image(s) generated based on the text prompt(s). As another example, the storage (540) can also store information about parameters of the target model and / or a learning domain of the target model. As yet another example, the storage (540) can store a program for obtaining synthetic images associated with a task of the target model and / or a program for optimizing the target model using the synthetic images. The program or data stored in the storage (540) can be loaded into the memory (500) before being executed by the processor (520). The storage (540) can store a file written in a programming language, and a program generated from the file by a compiler or the like can be loaded into the memory (500).

[0109] The input / output interface (560) provides an interface for data input / output with external devices. The external devices may include input devices such as a keyboard, mouse, or touch interface, and / or output devices such as a display. A user can trigger the execution of a program by the processor (520) through the input device, and check the results processed by the processor (520) through the output device.

[0110] The communication interface (580) provides access to an external network. For example, the computing device (50) may communicate with a server distributing a target model via the communication interface (580). As another example, the computing device (50) may communicate with a server executing an image generation model via the communication interface (580). As another example, the computing device (50) may transmit a calibration set to a terminal or distribute an optimized model via the communication interface (580).

[0111] While the above example illustrates a case where text prompts are generated in English, the present disclosure is not limited to this. That is, text prompts can be generated in various languages ​​supported by the image generation model.

[0112] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0113] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0114] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0115] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0116] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0117]

[0118] Statement regarding sponsored research or development

[0119] The present invention is a result of a research project (Project Unique Number: 1711195774, Subproject Number: 00228255, Ministry of Science and ICT, Project Management (Specialized) Agency: National IT Industry Promotion Agency, Research Project Name: Giant Artificial Neural Network Artificial Intelligence Semiconductor SW Technology Development (R&D), Research Project Name: Giant Artificial Neural Network Processing PIM-NPU Support System SW Technology Development, Contribution Rate: 1 / 1, Project Performing Agency Name: Sapion Korea Co., Ltd., Research Period: 2023.04.01. ~ 2023.12.31.).

[0120]

[0121] CROSS-REFERENCE TO RELATED APPLICATION

[0122] This patent application claims priority to Korean Patent Application No. 10-2023-0063824, filed on May 17, 2023, which is incorporated herein by reference in its entirety.

Claims

1. As a computer implementation method, A process of generating a text prompt based on a task-specific word pool associated with the task of an artificial neural network model; and A process of obtaining a calibration dataset for optimizing the artificial neural network model by inputting the above text prompt into a pre-trained image generation model. A method comprising:

2. In paragraph 1, A process of performing post-training quantization (PTQ) on the artificial neural network model based on the above calibration dataset. A method further comprising:

3. In paragraph 1, A process of performing pruning on the artificial neural network model based on the above calibration dataset. A method comprising:

4. In paragraph 1, A method wherein the text prompt is generated by applying a predefined rule to at least one word selected from the task-specific word pool.

5. In paragraph 1, The task of the above artificial neural network model is a classification task of classifying the class of an object included in an input image based on a predefined class set. A method wherein the task-specific word pool comprises a label for each of a plurality of classes constituting the predefined class set.

6. In paragraph 5, The process of generating the above text prompt is: The process of randomly selecting one or more labels from the task-specific word pool; and A process of generating a text prompt describing an image including one or more objects each corresponding to one or more of the selected labels using one or more of the selected labels. A method comprising:

7. In paragraph 6, The above text prompt is, comprising a first word combination representing the type of the image, one or more labels selected from the task-specific word pool, and one or more second word combinations modifying each label; A method wherein each of the first word combination and the one or more second word combinations comprises one or more words.

8. In paragraph 6, The above text prompt is, A first word combination representing the type of the image, a plurality of labels selected from the task-specific word pool, and a second word combination simultaneously modifying the plurality of labels, A method wherein the first word combination and the second word combination include one or more words.

9. In paragraph 1, The task of the above artificial neural network model is a segmentation task that divides an input image into multiple regions based on a predefined set of classes. A method wherein the task-specific word pool comprises a label for each of a plurality of classes constituting the predefined class set.

10. In paragraph 9, The process of generating the above text prompt is: The process of randomly selecting one label from the task-specific word pool; and A process of generating a text prompt describing the behavior or status of multiple objects corresponding to the same class using the above-selected label. A method comprising:

11. In paragraph 9, The above text prompt is, A method comprising a plural representation of a label selected from the task-specific word pool, and a phrase or clause describing a contingencies associated with the selected label.

12. In paragraph 1, The process of obtaining the above is, A process of performing the process of generating the above text prompt multiple times, inputting each of the obtained multiple text prompts into the image generation model, and collecting the synthetic images output by the image generation model corresponding to each text prompt to form the calibration dataset. A method comprising:

13. As a device for optimizing an artificial neural network model, A text generation module that generates a text prompt based on a task-specific word pool associated with the task of the artificial neural network model; and An image generation module that inputs the above text prompt into a pre-trained image generation model to obtain a calibration dataset for optimizing the artificial neural network model. A device comprising:

14. In paragraph 13, An optimization module that performs quantization or pruning after learning the artificial neural network model based on the above calibration dataset. A device further comprising:

15. In paragraph 13, A device wherein the text prompt is generated by applying a predefined rule to at least one word selected from the task-specific word pool.

16. In paragraph 13, The task of the above artificial neural network model is a classification task of classifying the class of an object included in an input image based on a predefined class set. A device wherein the task-specific word pool comprises a label for each of a plurality of classes constituting the set of predefined classes.

17. In paragraph 16, The above text generation module, A device that randomly selects one or more labels from the task-specific word pool and generates a text prompt describing an image containing one or more objects each corresponding to one or more of the selected labels using the one or more labels.

18. In paragraph 13, The task of the above artificial neural network model is a segmentation task that divides an input image into multiple regions based on a predefined set of classes. A device wherein the task-specific word pool comprises a label for each of a plurality of classes constituting the set of predefined classes.

19. In paragraph 18, The above text generation module, A device that randomly selects a label from the task-specific word pool and, using the selected label, generates a text prompt describing the actions or states of multiple objects corresponding to the same class.

20. A computer-readable recording medium having stored thereon a command, wherein the command, when executed by the computer, causes the computer to: The process of generating text prompts based on a task-specific word pool associated with the task of an artificial neural network model; and A process of obtaining a calibration dataset for optimizing the artificial neural network model by inputting the above text prompt into a pre-trained image generation model. A computer-readable recording medium that enables execution.

Citation Information

Patent Citations

  • Model pre-training method and device, equipment, storage medium and program product

    CN113962315A

  • Lightweight learning model

    JP7112802B1

  • Task specific processing of regulatory content

    US20220147814A1

  • Text Based Image Search

    US20220343626A1