Image classification model generation method and image processing system using image classification model

Abstract

The objective of the present invention is to provide: an image classification model generation method in which concepts are selected by optimally selecting concepts for image classification so as to minimize user intervention in model training and in which an efficient image classification model is provided according thereto; and an image classification system using the image classification model. The image classification model generation method according to the present invention comprises the steps of: predicting a plurality of test concepts for a test image by inputting the test image into an image classification model for predicting concepts of an image and classifying the class of the image on the basis of the predicted concepts; selecting, according to each concept selection criterion, a test concept to be corrected to improve the image classification model from among the plurality of test concepts by applying each test concept to each of a plurality of concept selection criteria for evaluating the concepts; improving the image classification model by correcting the test concept selected according to each concept selection criterion; and performing performance evaluation according to each concept selection criterion on the basis of the improved image classification model so as to select an optimal concept selection criterion from among the plurality of concept selection criteria, wherein the concept selection criterion includes at least one of a random scoring method, an uncertainty scoring method, a prediction loss scoring method, a contribution scoring method, a class change scoring method, and an entropy reduction degree scoring method.

Description

Method for generating an image classification model and an image processing system using an image classification model

[0001] The present invention relates to an image processing method and system, and more specifically, to an image processing method and system that performs image classification using a machine learning model.

[0002] In general, machine learning models are utilized in various fields. For example, machine learning models are used for image processing, such as image recognition or classification. Here, machine learning models can perform image processing through decision-making based on the complex interactions between many parameters.

[0003] Conventional image classification technology is disclosed in "Korean Patent Publication No. 10-2022-0131808 (Method and apparatus for generating an image classification model, Sep. 29, 2022)." The disclosed invention performs image classification based on a neural network model.

[0004] These machine learning models have limitations in that they are difficult to interpret due to complex interactions. Therefore, much research is being conducted on interpretable models to overcome the limitations of machine learning models.

[0005] For example, an interpretable model can perform preprocessing on an image using multiple concepts before performing image processing. Then, the interpretable model can perform image processing based on the results of the preprocessing. In this case, the interpretable model can provide the results of the preprocessing to the user, allowing the user to interpret the cause or basis of the image processing result.

[0006] Such interpretable models require expert intervention regarding conceptual prediction during the model generation process, and the reality is that significant expert intervention is required to improve model performance.

[0007] The objective of the present invention is to provide a method for generating an image classification model and an image classification system using the image classification model, which selects concepts using an optimal selection method for concepts regarding image classification to minimize user intervention in model training and thereby provide an efficient image classification model.

[0008] A method for generating an image classification model according to the present invention comprises the steps of: inputting a test image into an image classification model that predicts the concept of an image and classifies the class of the image based on the predicted concept to predict a plurality of test concepts for the test image; applying each test concept to each of a plurality of concept selection criteria for evaluating the concept to select a test concept among the plurality of test concepts to improve the image classification model according to each concept selection criterion; correcting the test concept selected according to each concept selection criterion to improve the image classification model; and performing a performance evaluation of each concept selection criterion based on the improved image classification model to select an optimal concept selection criterion among the plurality of concept selection criteria. The concept selection criterion includes at least one of a random scoring method, an uncertainty scoring method, a prediction loss scoring method, a contribution scoring method, a class change scoring method, and an entropy reduction degree scoring method.

[0009] In the step of selecting the optimal concept selection criteria, the performance evaluation of the concept selection criteria can be performed based on the number of times the test concept was corrected and the degree of improved performance of the image classification model.

[0010] In the step of improving the above image classification model, the above selected test concept can be improved by comparing it with the actual concept of the above test image and correcting the error of the above selected test concept.

[0011] In the step of improving the image classification model, the association between the selected test concepts is determined, and the error of the selected test concept can be corrected using either an individual method that individually corrects the selected test concepts based on the association between the selected test concepts, or a grouping method that groups related test concepts among the selected test concepts for correction.

[0012] The above optimal concept selection criterion may be the above uncertainty scoring method.

[0013] An image processing system according to the present invention is equipped with an image classification model generated by the image classification model generation method according to the present invention, inputs an image to be classified into the image classification model to predict the concept of the image to be classified, classifies the class of the image to be classified based on the predicted concept, and the image classification model uses an uncertainty score assignment method as an optimal concept selection criterion.

[0014] The image classification model generation method and image classification system using the image classification model according to the present invention include the effect of providing a cost-effective model that ensures model performance while minimizing expert intervention in model generation.

[0015] The technical effects of the present invention as described above are not limited to those mentioned above, and other unmentioned technical effects will be clearly understood by those skilled in the art from the description below.

[0016] FIG. 1 is a flowchart schematically illustrating an image classification system according to the present embodiment.

[0017] FIG. 2 is a configuration diagram showing an image classification system according to the present embodiment.

[0018] FIG. 3 is a flowchart illustrating an image processing method according to the present embodiment.

[0019] FIG. 4 is a flowchart showing the image classification model generation step among the image processing methods according to the present embodiment.

[0020] FIG. 5 is a flowchart showing the image class classification step of the image processing method according to the present embodiment.

[0021] Embodiments of the present invention will be described in detail below with reference to the attached drawings. However, the embodiments disclosed below are not limited to those disclosed below and may be implemented in various forms; the embodiments provided are merely intended to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention. The shapes of elements in the drawings may be exaggerated for clearer explanation, and elements indicated by the same reference numeral in the drawings represent the same element.

[0022] FIG. 1 is a flowchart showing an image processing system according to the present embodiment, and FIG. 2 is a configuration diagram showing an image processing system according to the present embodiment.

[0023] As illustrated in FIGS. 1 and 2, the image processing system (100) according to the present embodiment classifies an image (10) into a predetermined class (30) using an image classification model (200). The image classification model (200) classifies the class (30) of the image (10) using a plurality of concepts (50).

[0024] Here, the class (30) may be pre-configured and may be provided in multiple numbers. And the concept (50) may be sub-information of the class (30) for the image classification model (200) to classify the image (10) into a predetermined class (30).

[0025] For example, if the image (10) is of a bird, the class (30) may be information indicating the type of bird. And the concept (50) may be information that can derive the characteristics of the bird, such as the shape and / or color of the bird's wings, eyes, beak and legs. That is, the concept (50) may be applicable as a cause or basis for determining the class (30).

[0026] When an image (10) is input, the image classification model (200) predicts the concept (50) of the image (10). Then, the image classification model (200) determines the class of the image (10) based on the predicted concept (50). Here, the image classification model (200) can predict the concept (50) of the image (10) through an inference process based on a reference concept (51).

[0027] The image classification model (200) can provide the user (S) with the predicted concept (50) of the image (10) along with the determined class (30). Accordingly, the user (S) can use the predicted concept (50) to interpret it as a cause or basis for the determined class (30).

[0028] Additionally, the image processing system (100) can generate an image classification model (200) through interaction with a user (S). Here, the user (S) intervenes in the concept prediction process of the image classification model (200) to ensure the performance of the image classification model (200).

[0029] For example, the image processing system (100) may include a data input / output unit (110), a data storage unit (120), and a processing unit (130).

[0030] First, the data input / output unit (110) can receive data from the outside or output a predetermined analysis result. For example, the data input / output unit (110) may include an input unit (111) and an output unit (113). The input unit (111) can receive data from the outside. The input unit (111) can provide the input data to the storage unit (120) and the processing unit (130). The output unit (113) can output predetermined information by a control signal from the processing unit (130).

[0031] The user (S) can input data or check a predetermined processing result through the data input / output unit (110). The user (S) can check the image processing process and analysis results of the image classification model (200) through the data input / output unit (110). Additionally, the user (S) can intervene in the model creation process of the image classification model (200) through the data input / output unit (110).

[0032] Meanwhile, the data storage unit (120) can store a predetermined amount of data. The data storage unit (120) can store the aforementioned class (30) and concept (50). Additionally, the data storage unit (120) can store a preliminary image classification model (200), training data (121), and concept evaluation criteria (123) for generating an image classification model (200).

[0033] The image classification model (200) may be a machine learning model created by the user. For example, the prior image classification model (200) may be a machine learning model based on a neural network (NN).

[0034] The training data (121) may consist of multiple images. The training data (121) may be related to the images to be classified.

[0035] The concept evaluation criteria (123) are applied to each standard concept (51) to assign a score to each standard concept. The assigned score can be used when selecting the standard concept (51) to be used for image classification during the process of testing the image classification model (200).

[0036] For example, the concept evaluation criteria (123) may include a random scoring method, an uncertainty scoring method, a prediction loss scoring method, a contribution scoring method, a class change scoring method, and an entropy reduction degree scoring method.

[0037] The random scoring method assigns a score randomly to each reference concept (51). Accordingly, the reference concept (51) can be randomly selected through the random scoring method.

[0038] And the uncertainty scoring method calculates the degree of uncertainty of the predicted concept (50) when predicting the concept (50) of the image (10) with respect to the reference concept (50) and assigns a score. This uncertainty scoring method is defined by the following mathematical formula 1. In mathematical formula 1, H represents the entropy function, and This refers to the predicted concept (50) of an image (10) according to a reference concept (51) during testing of an image classification model (200). As shown below, the uncertainty score assignment method may be applied differently when the concept (51) is binary data.

[0039] [Mathematical Formula 1]

[0040] or

[0041] ( (If it is binary data)

[0042] And the predicted loss score assignment method assigns a score by calculating the degree of loss of the predicted concept of the image (10) relative to the reference concept (50) from the actual concept, that is, the concept set by the user. This predicted loss score assignment method is defined by the following mathematical formula 2. In mathematical formula 2, means the predicted concept (50) of the image (10) according to the reference concept (51) during testing of the image classification model (200), and means the actual concept (50) of the image (10).

[0043] [Mathematical Formula 2]

[0044]

[0045] And the contribution score assignment method calculates the degree to which the predicted concept (50) contributes to class (30) classification with respect to the reference concept (51) and assigns a score. The contribution score assignment method is defined by the following mathematical formula 3. In mathematical formula 3, means the predicted concept (50) of the image (10) according to the reference concept (51) during testing of the image classification model (200), and Is It refers to the output related to the nth target class. And ... refers to the number of classes.

[0046] [Mathematical Formula 3]

[0047]

[0048] And the class change score assignment method calculates the degree to which the class (30) to be determined can change due to the user (S)'s intervention regarding the concept (50) predicted for the reference concept (51) and assigns a score. The class change score assignment method is defined by the following mathematical formula 4. In mathematical formula 4, means the predicted concept (50) of the image (10) according to the reference concept (51) during testing of the image classification model (200), and represents Kullback-Leibler divergence. And silver It refers to user(S) intervention information that makes this 0, and silver This refers to the user(S) intervention information that makes this 1.

[0049] [Mathematical Formula 4]

[0050]

[0051] And the method for assigning a score for the degree of entropy reduction calculates the degree of entropy reduction of the predicted concept (50) that decreases when a user (S) intervenes in the predicted concept (50) of the image (10) with respect to the reference concept (51), and assigns a score. This method for assigning a score for the degree of entropy reduction is defined by Equation 5. In Equation 5, the predicted concept (50) of the image (10) according to the reference concept (51) during the test of the image classification model (200) is represented, and represents the entropy function. And silver It refers to user(S) intervention information that makes this 0, and silver This refers to the user(S) intervention information that makes this 1.

[0052] [Mathematical Formula 5]

[0053]

[0054] The concept evaluation criterion (123) can be selected as one of the aforementioned multiple concept evaluation methods and used to select the standard concept (51). For example, the concept evaluation criterion (123) can be selected during the process of testing the image classification model (200).

[0055] For example, the user (S) can select a concept evaluation method by calculating the intervention cost of the standard concept (51) selected for each concept evaluation method through a preliminary experiment on the image classification model (200).

[0056] Specifically, the user (S) inputs a test image into a trained image classification model (200) for a preliminary experiment. At this time, the user (S) may input the actual class of the test image and the actual concept of each reference concept (51) together. The user (S) predicts the concept and class of the test image based on multiple reference concepts (51). The user (S) applies the predicted concept, predicted class, actual concept, and actual class of the test image to each concept evaluation method and assigns a score to the reference concept (51) using each concept evaluation method.

[0057] The user (S) selects a reference concept (51) based on the assigned score. Then, the user (S) searches for errors in the predicted concept by comparing the predicted concept predicted by the selected reference concept (51) with the actual concept. The user (S) can perform intervention on the concept prediction of the image classification model (200) by replacing the predicted concept classified as an error with the actual concept. At this time, the user (S) calculates the number of interventions and can calculate the error reduction rate of the image classification model (200) according to the number of interventions.

[0058] In this way, the user (S) judges the efficiency of each concept evaluation method and selects the most efficient concept evaluation method among multiple concept evaluation methods. At this time, the user (S) can select the optimal concept evaluation method that has the fewest number of interventions in concept prediction and the highest error reduction rate of the image classification model (200).

[0059] Accordingly, the optimal concept evaluation method is applied to improve the performance of the image classification model (200), thereby minimizing user (S) intervention during the model generation process. For example, the optimal concept evaluation method may be an uncertainty score assignment method.

[0060] Meanwhile, the processing unit (130) can perform predetermined computation processing and data processing. The processing unit (130) generates an image classification model (200) and classifies an image (10) into a predetermined class (30) using the generated image classification model (200).

[0061] Meanwhile, the operation of the image processing system (100) will be explained below with reference to the attached FIGS. 3 to 5.

[0062] FIG. 3 is a flowchart illustrating an image processing method according to the present embodiment, FIG. 4 is a flowchart illustrating an image classification model generation step among the image processing methods according to the present embodiment, and FIG. 5 is a flowchart illustrating an image class classification step among the image processing methods according to the present embodiment.

[0063] As illustrated in FIGS. 3 to 5, the image processing system (100) according to the present embodiment can classify the class (30) of an image to be classified through an image classification model generation step (S100) and an image class classification step (S200).

[0064] First, in the step of creating an image classification model (S100), the user (S) can obtain a plurality of standard concepts (51) for classifying the image to be classified (S110).

[0065] And the image processing system (100) trains the image classification model (200) based on training data (121) and multiple reference concepts (51) (S120).

[0066] Then, the image processing system (100) inputs the test image (11) into the image classification model (200) and performs a test on the image classification model (200) (S130). At this time, the image processing system (100) can input the actual concept and the actual class of the test image together. Then, the image classification model (200) predicts the predicted concept and the predicted class of the test image (11) using each reference concept (51).

[0067] Then, the image classification model (200) applies the prediction concept and prediction class of the test image (11) to the optimal concept evaluation method and assigns a score to the reference concept (51). Then, based on the assigned score, the image classification model (200) selects a reference concept (51) among the reference concepts (51) for which the user (S) will intervene to improve the performance of the image classification model (200) (S140).

[0068] The image processing system (100) provides the user (S) with a predicted concept predicted by a selected reference concept (51). The user (S) detects an error by comparing the predicted concept with the actual concept of the test image (11) and corrects the detected error to the actual concept.

[0069] At this time, the user (S) can correct errors in an individual manner or a grouping manner depending on the association of the reference concepts (51). The individual method means performing correction for each reference concept (51). The grouping method means that the user (S) groups the related reference concepts (51) and performs correction on the grouped concept group.

[0070] Additionally, the user (S) can set the number of corrections for each image (10) according to the data accessibility of the image (10). The number of corrections can be set according to the characteristics of the image (10). Here, the number of corrections can be set in a single method or a batch method.

[0071] The single method means correcting each test image (10) the same number of times. The single method can be useful in online systems where data comes in sequentially and in environments where the number of experts is limited.

[0072] The batch method means setting the number of corrections before correction and distributing the number of corrections for each image (10) according to the set number of corrections to perform corrections. This batch method can be effective in cases where predictions are unevenly made, such as in cases where concept prediction is easy. Additionally, the batch method can be used to maximize overall work performance.

[0073] Accordingly, the user (S) can perform correction in an appropriate manner by considering the relevance of the reference concept (51) and the data accessibility of the test image (11).

[0074] The image processing system (100) calculates the performance of an image classification model (200) that has been corrected for concept prediction, and when the calculated performance reaches a predetermined value, it can be used as the final image classification model (S150).

[0075] Meanwhile, in the image class classification step (S200), the image processing system (100) can perform image classification for the target image based on the final image classification model.

[0076] First, the image processing system (100) inputs the image to be classified into the final image classification model (205) (S210). The final image classification model performs concept prediction based on a reference concept (51) for the image to be classified (S220). Then, the final image classification model determines the class (30) of the image to be classified based on the result of each concept prediction, that is, the concept of the image to be classified (S230).

[0077] And the image processing system (100) can provide the results of each concept prediction and class (30) for the image to be classified to the user (S).

[0078] In this way, the image processing system according to the present invention enables cost-effective model learning by applying an optimal concept selection method to the concept for classifying images into a predetermined class.

[0079] Accordingly, the image classification model generation method and image classification system using the image classification model according to the present invention include the effect of providing a cost-effective model that ensures model performance while minimizing expert intervention in model generation.

[0080] An embodiment of the present invention described above and illustrated in the drawings should not be interpreted as limiting the technical scope of the present invention. The scope of protection of the present invention is limited only by the matters described in the claims, and a person skilled in the art may modify or change the technical scope of the present invention in various forms. Accordingly, such modifications and changes will fall within the scope of protection of the present invention insofar as they are obvious to a person skilled in the art.

Claims

1. A step of predicting multiple test concepts for a test image by inputting a test image into an image classification model that predicts the concept of an image and classifies the class of the image based on the predicted concept; A step of applying each test concept to each of the plurality of concept selection criteria for evaluating the above concept, and selecting a test concept to be corrected among the plurality of test concepts to improve the image classification model according to each of the above concept selection criteria; A step of improving the image classification model by correcting the test concept selected according to each of the above concept selection criteria; and It includes a step of selecting the optimal concept selection criterion among multiple concept selection criteria by performing a performance evaluation of each concept selection criterion based on the improved image classification model above. The above concept selection criteria are A method for generating an image classification model characterized by including at least one of a random scoring method, an uncertainty scoring method, a prediction loss scoring method, a contribution scoring method, a class change scoring method, and an entropy reduction degree scoring method.

2. In Paragraph 1, In the step of selecting the optimal concept selection criteria mentioned above, A method for generating an image classification model characterized by performing a performance evaluation of the concept selection criteria based on the number of times the above test concept has been corrected and the degree of improved performance of the above image classification model.

3. In Paragraph 1, In the step of improving the above image classification model, A method for generating an image classification model characterized by comparing the selected test concept with the actual concept of the test image to correct and improve errors in the selected test concept.

4. In Paragraph 3, In the step of improving the above image classification model, Determine the association between the above-mentioned selected test concepts, and A method for generating an image classification model characterized by correcting errors in selected test concepts using either an individual method of individually correcting selected test concepts according to the association between selected test concepts, or a grouping method of grouping related test concepts among selected test concepts to correct them.

5. In Paragraph 3, In the step of improving the above image classification model, Determine the data accessibility of the above test image, and A method for generating an image classification model, characterized by setting the number of corrections for the test images according to the data accessibility of the test images, using either a single method of correcting each of the test images an equal number of times or a batch method of correcting each test image concept by distributing a preset number of times for all of the test images.

6. In Paragraph 1, The above optimal concept selection criteria is An image processing method characterized by the above-mentioned uncertainty scoring method.

7. An image classification model generated by the method described in claim 1 is loaded, and an image to be classified is input into the image classification model to predict the concept of the image to be classified, and classifies the class of the image to be classified based on the predicted concept, and The above image classification model is An image processing system characterized by using an uncertainty scoring method as the optimal concept selection criterion.

Images