Learning data selection support device, learning data selection method, determination accuracy improvement method, and learning data selection support program
The training data selection support device enhances the efficiency of selecting appropriate training data by using trained vectorization units to increase text-image similarity, effectively rejecting inappropriate pairs and reducing manual verification needs.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2026-03-05
AI Technical Summary
Conventional techniques require significant effort and specialized skills to select appropriate image-text pairs from a vast number of training data candidates, as determining appropriateness is labor-intensive and often beyond the capabilities of many doctors.
A training data selection support device comprising an input unit, text and image vectorization units, and a determination unit that leverages trained vectorization units to increase similarity between text and image vectors for correct pairs, allowing efficient rejection of inappropriate data.
The device efficiently selects appropriate training data by rejecting obviously inappropriate candidates, reducing the need for manual verification of difficult cases, thereby facilitating efficient selection from a large number of image-text pairs.
Smart Images
Figure 2026036505000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a training data selection support device, a training data selection method, a determination accuracy improvement method, and a training data selection support program for selecting training data from a huge number of training data candidates. [Background technology]
[0002] CLIP (Connecting Text and Images), described in Non-Patent Documents 1 and 2, is known as a multimodal model of language and images. Non-Patent Document 1 explains that "The basic idea of CLIP is to learn good language and image representations (embeddings) through contrastive learning of images paired with linguistic information (text)." "During training, mini-batches of text and image pairs are created, and the cosine similarity is calculated for each text and image representation. The training is performed so that it increases for correct pairs and decreases for incorrect pairs." Non-Patent Document 2 also introduces one of CLIP's features as a "natural language supervised image classification model that allows users to freely set categories," explaining that "the training data is not composed of a combination of general images and labels (which has a low degree of freedom), but rather images and text describing the images (which has a high degree of freedom). This means that the categories that can be set as labels are not limited, and the degree of freedom in setting categories is improved." Non-Patent Document 3 describes the training of an image model that can be converted from natural language expressions. [Prior art documents] [Non-patent literature]
[0003] [Non-Patent Document 1] Tatsuya Matsushima, "CLIP: A Multimodal Foundation Model of Language and Image," last updated December 2, 2022, [Retrieved July 18, 2024], Internet<https: / / trail.t.u-tokyo.ac.jp / ja / blog / 22-12-02-clip / > . [Non-patent document 2] Norihisa Kobayashi, "Learning Transferable Visual Models From Natural Language Supervision", DL Reading Group, January 15, 2021, [Retrieved July 18, 2024], Internet <https: / / www.slideshare.net / slideshow / dllearning-transferable-visual-models-from-natural-language-supervision / 241356871> . [Non-patent document 3] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever, “Learning Transferable Visual Models From Natural Language Supervision”, arxiv, Submitted on 26 Feb 2021, [Retrieved July 24, 2024], Internet<https: / / arxiv.org / abs / 2103.00020> . Summary of the Invention [Problem to be solved by the invention]
[0004] Conventional techniques have been shown to train systems so that cosine similarity increases for correct image-text pairs and decreases for incorrect pairs. However, to achieve this training effect, it is necessary to train using correct pairs. For example, when selecting which image-text pairs are appropriate and which are inappropriate from the vast number of image-text pairs held by hospitals, a huge amount of effort is required, and there is a problem that few doctors have the skills to determine whether a pair is appropriate or inappropriate. Therefore, an object of the present invention is to provide a technology for efficiently selecting appropriate training data from a vast number of image-text pairs. [Means for solving the problem]
[0005] The training data selection support device of the present invention comprises an input unit, a text information vectorization unit, an image information vectorization unit, and a determination unit. The input unit acquires pairs of text information and image information as training data candidates. The text information vectorization unit vectorizes the text information of the training data candidates and outputs text vectors. The image information vectorization unit vectorizes the image information of the training data candidates and outputs image vectors. The text information vectorization unit and the image information vectorization unit are trained using image information paired with previously collected text information so that in the case of a correct pair of text information and image information, the similarity between the text vector and the image vector is large. The determination unit determines whether to accept or reject the training data candidate based on the text vector and image vector for the training data candidate. [Effects of the Invention]
[0006] According to the training data selection support device of the present invention, image information paired with previously collected text information is used, and in the case of a correct text and image pair, a text information vectorization unit and an image information vectorization unit are used that have been trained to increase the similarity between the text vector and the image vector. Then, based on the text vector and image vector for the training data candidate, it is determined whether or not the training data candidate is acceptable. Since the training data selection support device can reject obviously inappropriate training data candidates, a person need only check training data candidates that are difficult to determine whether or not they are appropriate pairs. Therefore, appropriate training data can be efficiently selected from a huge number of image and text pairs. [Brief explanation of the drawings]
[0007] [Figure 1] FIG. 1 is a diagram showing an example of the functional configuration of a learning data selection support device according to the present invention. [Figure 2] FIG. 10 is a diagram showing an example of a processing flow of a training data selection method using the training data selection support device. [Figure 3] FIG. 10 is a diagram showing an example of a processing flow of a method for improving the determination accuracy of a learning data selection support device. [Figure 4] FIG. 2 is a diagram showing an example of the functional configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION
[0008] Hereinafter, embodiments of the present invention will be described in detail. Components having the same functions are given the same numbers, and duplicated explanations will be omitted. [Example]
[0009] Fig. 1 shows an example of the functional configuration of a training data selection support device of the present invention. Fig. 2 shows an example of the processing flow of a training data selection method using the training data selection support device. The training data selection support device 100 includes an input unit 110, a text information vectorization unit 120, an image information vectorization unit 130, a determination unit 140, and a recording unit 190. The training data selection support device 100 may also include a determination confirmation unit 150.
[0010] The input unit 110 acquires pairs of text information and image information as training data candidates (S110). In the present invention, it is assumed that there will be a huge amount of training data candidates. A huge amount of training data candidates may be recorded in advance in the recording unit 190. The training data candidates are, for example, pairs of text information and image information related to medical diagnoses. The text information related to medical diagnoses is a document written by a doctor or the like based on the image information. Medical-related image information may include, but is not limited to, X-rays, CT (Computed Tomography) images, MRI (Magnetic Resonance Imaging) images, electrocardiogram waveforms, and table data displaying blood components in a tabular format. Furthermore, the training data candidates do not need to be limited to medical-related data.
[0011] The text information vectorization unit 120 vectorizes the text information of the training data candidates and outputs text vectors (S120). The text information vectorization unit 120 may also output a classification result of the text information of the training data candidates. For example, when the training data candidates are pairs of text information and image information related to medical diagnoses, the text information of the training data candidates may contain a diagnosis result (such as a disease name). In such a case, the text information vectorization unit 120 may also output a classification result based on the diagnosis result included in the text information of the training data candidates. Possible classifications include, but are not limited to, the name of the disease (including no problem), treatment method, cause, etc.
[0012] The image information vectorization unit 130 vectorizes the image information of the training data candidates and outputs image vectors (S130). The text information vectorization unit 120 and the image information vectorization unit 130 are trained using image information paired with pre-collected text information so that the similarity between the text vector and the image vector increases when the text information and image information are paired correctly. The similarity may be, for example, cosine similarity, but is not limited to this and may be any similarity that achieves the effects of the present invention.
[0013] The determination unit 140 determines whether to accept or reject a training data candidate based on the text vector and image vector for the training data candidate (S140). More specifically, the determination unit 140 may determine that the training data candidate is accepted when the similarity between the text vector and image vector for the training data candidate is greater than a predetermined value. The role of the training data selection support device 100 of the present invention is to reduce the labor required for a person to select training data candidates suitable as training data. Therefore, the training data selection support device 100 is required to exclude (delete) training data candidates that are clearly inappropriate as training data. "Accepting" means retaining training data candidates that are not clearly inappropriate as training data. "Rejecting" means removing training data candidates that are clearly inappropriate as training data.
[0014] The text information vectorization unit 120 may also output a classification result for the text information of the training data candidate. The text information may contain information about the classification result. For example, if a photo of an animal is used as image information and the paired text information contains "panda," the animal may be classified as "panda." In this case, a "predetermined value" for determining whether to accept or reject the classification result may be defined for each classification result. For example, the "predetermined value" when the text information vectorization unit 120 outputs a classification result of "panda" may be set to a different value than the "predetermined value" when the text information vectorization unit 120 outputs a classification result of "rhinoceros." This is because there may be a bias in the degree of learning between the text information vectorization unit 120 and the image information vectorization unit 130, and the accuracy of vectorization may differ for each classification (e.g., animal type). In such a case, the determination unit 140 may determine that the training data candidate is accepted if the similarity between the text vector and image vector for the training data candidate is greater than a value defined for each classification result. Since the value can be adjusted for each classification result, more desirable judgments can be expected.
[0015] The same applies when the training data candidates are pairs of text information and image information related to medical diagnoses. The text information vectorization unit 120 may also output classification results based on the diagnosis results contained in the text information of the training data candidates. For example, classifications may be based on information about the disease name, such as "pneumonia" or "no abnormalities," or on treatment methods, such as the contents of intravenous drips and prescriptions. In such cases, the determination unit 140 may determine that a training data candidate should be adopted if the similarity between the text vector and image vector for that training data candidate is greater than a value determined for each classification result. Since the value can be adjusted for each classification result, more desirable determinations can be expected.
[0016] <Explanation of the method for selecting training data> Next, with reference to FIG. 2, the processing flow of the training data selection method using the training data selection support device 100 will be described. The training data selection support device 100 determines whether a training data candidate is adopted or rejected (candidate determination step S100). The above-mentioned steps S110 to S140 are combined into one step, the candidate determination step S100. Next, a person (e.g., a doctor in the case of medical information) determines whether the training data candidate determined to be adopted in the candidate determination step S100 is appropriate (determination step S101). The training data candidate determined to be appropriate in the determination step S101 (if Yes) is recorded as training data (training data recording step S190). The recording medium may be the recording unit 190 of the training data selection support device 100 or a recording medium of a different device. If there are other training data candidates, these processes can be repeated (S102).
[0017] The training data selection support device 100 uses image information paired with pre-collected text information, and employs a text information vectorization unit 120 and an image information vectorization unit 130 trained to maximize the similarity between the text vector and the image vector when the text information and image information pair is correct. Then, based on the text vector and image vector for the training data candidate, it determines whether or not the training data candidate is acceptable. Since the training data selection support device 100 can reject obviously inappropriate training data candidates, a human need only check training data candidates whose appropriateness is difficult to determine. This allows for efficient selection of appropriate training data from a vast number of image-text pairs.
[0018] <Explanation of methods for improving judgment accuracy> Next, a method for improving the determination accuracy of the training data selection support device 100 will be described. FIG. 3 shows an example of a processing flow of the method for improving the determination accuracy of the training data selection support device. The text information vectorization unit 120 and the image information vectorization unit 130 are trained using image information paired with previously collected text information so that, in the case of a correct pair of text information and image information, the similarity between the text vector and the image vector is large. However, it is considered that the initial settings are not sufficient to achieve sufficient determination accuracy. The role of the training data selection support device 100 of the present invention is to reduce the labor required for a person to select suitable training data from among the training data candidates. Therefore, the training data selection support device 100 is required to exclude (delete) training data candidates that are clearly inappropriate as training data.
[0019] When the initial settings do not provide sufficient judgment accuracy, the training data selection support device 100 may also include a judgment confirmation unit 150. The judgment confirmation unit 150 displays the training data candidates and the judgments, obtains the confirmation results for the judgments, and records the training data candidates that have been confirmed to be misjudged and the similarities between the text vectors and image vectors for the training data candidates in the recording unit 190 (S150). In other words, a person judges whether the training data candidates and the judgments displayed by the judgment confirmation unit 150 are valid and obtains the results.
[0020] There are two types of judgment errors: errors in rejecting a training data candidate that should have been judged as acceptable, and errors in adopting a training data candidate that should have been judged as unacceptable. What is required of the training data selection support device 100 is to exclude (delete) training data candidates that are clearly inappropriate as training data. Therefore, it is also important to reduce errors in rejecting a training data candidate that should have been judged as acceptable.
[0021] Therefore, the method for improving the determination accuracy of the training data selection support device 100 may be processed as follows. First, the training data selection support device 100 determines whether a training data candidate is to be adopted or rejected (candidate determination step S100). The candidate determination step S100 is a step that combines the above-mentioned steps S110 to S140. Next, a person confirms the determination of the training data candidate determined in the candidate determination step (determination confirmation step S150). Here, both errors in rejecting a training data candidate that should have been determined to be adopted and errors in adopting a training data candidate that should have been determined to be rejected are confirmed. In the confirmation, whether the determination of adoption or rejection was appropriate is more important than whether the determination was appropriate.
[0022] Next, the training data selection support device 100 records the training data candidates that were confirmed to be erroneous in the judgment confirmation step S150 and the similarity between the text vectors and image vectors for the training data candidates (incorrect judgment recording step S191). If there are other training data candidates, the process is repeated (S102). After the repeated process is completed, additional learning is performed on the text information vectorization unit 120 and the image information vectorization unit 130, or the judgment criteria of the judgment unit 140 are adjusted (improvement step S105) so that the judgment by the training data selection support device 100 becomes correct.
[0023] If an erroneous determination occurs due to low vectorization accuracy of the text information vectorization unit 120 or the image information vectorization unit 130, additional learning of the text information vectorization unit 120 or the image information vectorization unit 130 that is causing the determination can be performed. Furthermore, the determination criteria of the determination unit 140 can be adjusted, for example, by adjusting a predetermined value. In cases where the text information vectorization unit 120 may also output classification results for the text information of the training data candidates, the value determined for each classification result of the determination unit 140 can also be adjusted. By improving the determination accuracy of the training data selection support device 100 in this way, appropriate training data can be efficiently selected from an even larger number of image-text pairs.
[0024] For example, if overall accuracy improvement is desired, retraining or additional training is performed on the text information vectorization unit 120 and the image information vectorization unit 130, and the judgment criteria of the judgment unit 140 are adjusted. If the accuracy of either the vectorization by the text information vectorization unit 120 or the vectorization by the image information vectorization unit 130 is low, retraining or additional training can be performed on only the vectorization with lower accuracy. The decision whether to improve overall accuracy or improve only the one with lower accuracy can be made manually. Note that the judgment confirmation unit 150 may evaluate the accuracy of the vectorization based on the text vector and image vector when an incorrect judgment is confirmed, and record the text vector, image vector, and evaluation results in the recording unit 190. In this case, the training data selection support device 100 may determine or suggest whether to improve overall accuracy or improve only the one with lower accuracy based on the evaluation results. Other special methods for additional training by the text information vectorization unit 120 and the image information vectorization unit 130 include the following. First, the text vectors and image vectors of data that have already been trained are recorded in the recording unit 190. Note that only a representative portion of the text vectors and image vectors may be recorded in the recording unit 190. Next, the text information and image information of new training data are vectorized by the text information vectorization unit 120 and the image information vectorization unit 130. Here, the distance between the vector group recorded in the recording unit 190 and the new training data is calculated to evaluate whether or not it corresponds to an outlier. If the text vector of the new training data corresponds to an outlier, the text information vectorization unit 120 is trained additionally. Conversely, if the image vector of the new training data corresponds to an outlier, the image information vectorization unit 130 is trained additionally. If both correspond to outliers, the text information vectorization unit 120 and the image information vectorization unit 130 are trained additionally. Existing outlier detection methods can be used to determine whether or not they correspond to outliers. [Variation 1]
[0025] Image information can also be classified based on the type of image information it contains. For example, in the case of a pair of text information and image information related to a medical diagnosis, the image information can be classified based on the type of image information, such as an X-ray, a CT image, an MRI image, an electrocardiogram waveform, or table data displaying diagnostic results in a tabular format. It may be desirable to set a "predetermined value" for each classification to determine whether to accept or reject the image. However, classification does not need to be limited to the type of image information, and other classifications may be used as long as they are easy to determine from the image information and it is useful to set a "predetermined value" for each classification.
[0026] An example of the functional configuration of the training data selection support device is the same as that shown in Figure 1, an example of the processing flow of a training data selection method using the training data selection support device is the same as that shown in Figure 2, and an example of the processing flow of a method for improving the judgment accuracy of the training data selection support device is the same as that shown in Figure 3.
[0027] In this modification, the image information vectorization unit 130 outputs the classification result in addition to the image vector of the image information of the training data candidate (S130). The determination unit 140 determines that the training data candidate should be adopted when the similarity between the text vector and the image vector for the training data candidate is greater than a value determined for each classification result (S140). For example, a "predetermined value" can be set for each type of image information, which is expected to result in more desirable adoption / rejection decisions.
[0028] The text information vectorization unit 120 may also output the classification results contained in the text information of the training data candidates. For example, if the text information contains the word "X-ray," the classification results can also be output from the text information. In such a case, the determination unit 140 may determine that the training data candidates should be adopted if the classification results of the text information vectorization unit 120 and the image information vectorization unit 130 for the training data candidates are the same and the similarity between the text vector and image vector for the training data candidate is greater than a value determined for each classification result.
[0029] In this modification, the training data selection support device 100 may also include a judgment confirmation unit 150. To improve the judgment accuracy, additional learning may be performed on the text information vectorization unit 120 and the image information vectorization unit 130, or the values defined for each classification result of the judgment unit 140 may be adjusted so that the judgment by the judgment unit 140 is correct. By improving the judgment accuracy of the training data selection support device 100 in this way, appropriate training data can be efficiently selected from an even larger number of image and text pairs.
[0030] [Processor, program, recording medium] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes programs stored in memory.
[0031] In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed herein or any hardware known to be programmed to realize or perform the described functions.
[0032] If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.
[0033] The various processes described above can be implemented by loading a program that executes each step of the above method into the recording unit 2020 of the computer 2000 shown in Figure 4, and operating the control unit 2010, input unit 2030, output unit 2040, display unit 2050, etc.
[0034] The program describing the processing contents can be recorded on a computer-readable recording medium, which may be, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, a semiconductor memory, or any other suitable recording medium.
[0035] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.
[0036] A computer that executes such a program may first temporarily store the program recorded on a portable recording medium or transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process in accordance with the read program. Alternatively, the computer may read the program directly from a portable recording medium and execute the process in accordance with the program. Furthermore, the computer may execute the process in accordance with the program each time a program is transferred from a server computer to the computer. The server computer may not transfer the program to the computer, but may instead execute the process through a so-called ASP (Application Service Provider) service, which realizes the processing function by issuing an execution instruction and obtaining the results. Furthermore, the server computer may execute the process on a terminal using a so-called SaaS (Software as a Service) service, which allows users to use part of the server computer along with the program. In this embodiment, the program includes information used for computer processing that is equivalent to a program (such as data that is not a direct instruction to the computer but has properties that define computer processing).
[0037] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware. [Explanation of symbols]
[0038] 100 Learning data selection support device 110 Input unit 120 Text information vectorization unit 130 Image information vectorization unit 140 Judgment section 150 Judgment confirmation section 190 Recording Department
Claims
1. an input unit that acquires pairs of text information and image information as training data candidates; a text information vectorization unit that vectorizes text information of the learning data candidates and outputs text vectors; an image information vectorization unit that vectorizes the image information of the learning data candidates and outputs image vectors; a determination unit that determines whether a training data candidate is adopted or rejected based on the text vector and the image vector for the training data candidate; Equipped with The text information vectorization unit and the image information vectorization unit are trained using image information paired with previously collected text information so that the similarity between the text vector and the image vector increases when the text information and image information are paired correctly. A training data selection support device characterized by:
2. 2. The training data selection support device according to claim 1, The determination unit determines that the candidate learning data is adopted when the similarity between the text vector and the image vector for the candidate learning data is greater than a predetermined value. A training data selection support device characterized by:
3. 2. The training data selection support device according to claim 1, When the text information vectorization unit also outputs a classification result of the text information of the learning data candidates, The determination unit determines that the candidate learning data is adopted when the similarity between the text vector and the image vector is greater than a value determined for each classification result. A training data selection support device characterized by:
4. 2. The training data selection support device according to claim 1, the training data candidates are pairs of text information and image information related to medical diagnoses; When the text information vectorization unit also outputs a classification result based on a diagnosis result included in the text information of the learning data candidate, The determination unit determines that the candidate learning data is adopted when the similarity between the text vector and the image vector is greater than a value determined for each classification result. A training data selection support device characterized by:
5. 2. The training data selection support device according to claim 1, the training data candidates are pairs of text information and image information related to medical diagnoses; The image information vectorization unit also outputs a classification result of the image information of the learning data candidates, The determination unit When the similarity between the text vector and the image vector for the training data candidate is greater than a value determined for each classification result, the candidate is determined to be adopted. A training data selection support device characterized by:
6. 6. The training data selection support device according to claim 5, When the text information vectorization unit also outputs the classification result included in the text information of the training data candidate, The determination unit When the classification results of the text information vectorization unit and the image information vectorization unit for the training data candidate are the same, and the similarity between the text vector and the image vector for the training data candidate is greater than a value determined for each classification result, the training data candidate is determined to be adopted. A training data selection support device characterized by:
7. 7. The training data selection support device according to claim 1, Furthermore, it is provided with a judgment confirmation unit and a recording unit, The judgment confirmation unit displays the learning data candidates and the judgment, acquires a confirmation result for the judgment, and records the learning data candidates that have been confirmed to be misjudged and the similarity between the text vector and the image vector for the learning data candidates in the recording unit. A training data selection support device characterized by:
8. A training data selection method using the training data selection support device according to any one of claims 1 to 6, comprising: a candidate determination step in which the learning data selection support device determines whether a learning data candidate is adopted or rejected; a determining step in which a person determines whether the learning data candidates determined to be adopted in the candidate determining step are appropriate; a learning data recording step of recording the learning data candidates determined to be appropriate in the determining step as learning data; A method for selecting training data to perform the above.
9. A method for improving the judgment accuracy of a training data selection support device according to any one of claims 1 to 6, comprising: a candidate determination step in which the learning data selection support device determines whether a learning data candidate is adopted or rejected; a determination confirmation step in which a person confirms the determination of the learning data candidate determined in the candidate determination step; an erroneous determination recording step in which the training data selection support device records the training data candidate confirmed to be an erroneous determination in the determination confirmation step and the similarity between the text vector and the image vector for the training data candidate; An improvement step of performing additional learning of the text information vectorization unit and the image information vectorization unit or adjusting the judgment criteria of the judgment unit so that the judgment of the learning data selection support device becomes correct. A method for improving judgment accuracy by performing the above.
10. A method for improving the judgment accuracy of a training data selection support device according to any one of claims 1 to 6, comprising: a candidate determination step in which the learning data selection support device determines whether a learning data candidate is adopted or rejected; a determination confirmation step in which a person confirms the determination of the learning data candidate determined in the candidate determination step; an erroneous determination recording step in which the training data selection support device records the training data candidate confirmed to be an erroneous determination in the determination confirmation step and the similarity between the text vector and the image vector for the training data candidate; An improvement step of performing additional learning on the text information vectorization unit or the image information vectorization unit so that the judgment of the learning data selection support device becomes correct. A method for improving judgment accuracy by performing the above.
11. A method for improving the judgment accuracy of a training data selection support device according to any one of claims 3 to 6, comprising: a candidate determination step in which the learning data selection support device determines whether a learning data candidate is adopted or rejected; a determination confirmation step in which a person confirms the determination of the learning data candidate determined in the candidate determination step; an erroneous determination recording step in which the training data selection support device records the training data candidate confirmed to be an erroneous determination in the determination confirmation step and the similarity between the text vector and the image vector for the training data candidate; An improvement step of adjusting values determined for each classification result of the determination unit so that the determination of the learning data selection support device becomes correct. A method for improving judgment accuracy by performing the above.
12. A learning data selection support program for causing a computer to function as the learning data selection support device according to any one of claims 1 to 6.