Learning data collection device, learning data collection method, and learning data collection program

Abstract

A learning data collection device (1) comprises: a learning device (20) that generates a trained model (31) by training using labeled learning data (T10); an inference unit (30) that detects an object captured in a target image of unlabeled data (T40) using the trained model (31) and outputs a detection result (D31); an other-viewpoint output generation unit (50) that uses an other-viewpoint trained model (51) having a structure different from that of the trained model (31) to generate information on the object as an other-viewpoint output (D51); a calculation unit (60) that calculates a matching degree (C60) between the detection result (D31) and the other-viewpoint output (D51); and a collection unit (70) that collects, as collected data (C70), unlabeled data having a matching degree (C60) lower than a reference value.

Description

Learning Data Collection Device, Learning Data Collection Method, and Learning Data Collection Program

[0001] The present disclosure relates to a learning data collection device, a learning data collection method, and a learning data collection program.

[0002] Devices have been proposed that can analyze video, identify objects in the video, and express the objects using natural language expressions (annotation) by using an object detection model as a machine learning model (see, for example, Patent Document 1).

[0003] Japanese Unexamined Patent Application Publication No. 2020-173791

[0004] However, Patent Document 1 does not propose a method for collecting learning data for improving the performance of an object detection model. Therefore, it is difficult to improve the performance of the object detection model in the device described in Patent Document 1.

[0005] An object of the present disclosure is to provide a learning data collection device, a learning data collection method, and a learning data collection program that enable collection of learning data for improving the performance of an object detection model.<

[0006] The learning data collection device of the present disclosure includes a learning device that acquires labeled learning data including learning images and labels from a learning data set and generates a learned model by learning using the labeled learning data; an inference unit that acquires unlabeled data including a target image from an image data set, detects an object appearing in the target image using the learned model, and outputs a detection result; an other-viewpoint output generation unit that acquires the unlabeled data and generates information about the object included in the unlabeled data as an other-viewpoint output using an other-viewpoint learned model having a structure different from that of the learned model; a calculation unit that calculates a degree of agreement indicating the degree of agreement between the detection result and the other-viewpoint output; and a collection unit that collects the unlabeled data having a degree of agreement lower than a predetermined reference value as collection data.

[0007] The training data collection method of this disclosure is a method performed by a training data collection device and is characterized by comprising the steps of: acquiring labeled training data including training images and labels from a training dataset, and generating a trained model by training using the labeled training data; acquiring unlabeled data including target images from an image dataset, detecting objects in the target images using the trained model, and outputting a detection result; acquiring the unlabeled data, and generating information about objects contained in the unlabeled data as an extra-viewpoint output using an extra-viewpoint trained model having a different structure from the trained model; calculating a degree of agreement indicating the degree of agreement between the detection result and the extra-viewpoint output; and collecting the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.

[0008] The learning data collection device, learning data collection method, and learning data collection program of this disclosure can be used to collect learning data for improving the performance of an object detection model.

[0009] This is a block diagram schematically showing the configuration of the learning data collection device according to Embodiment 1. This is a diagram showing an example of the hardware configuration of the learning data collection device according to Embodiment 1. This is a flowchart showing the operation of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (1) showing example operation E1 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (2) showing example operation E1 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (3) showing example operation E1 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (1) showing example operation E2 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (part 2) showing an example of operation E2 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams showing an example of operation E3 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams showing an example of operation E4 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams showing an example of operation E5 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams showing an example of operation E6 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. (A) to (D) are explanatory diagrams (part 1) showing an example of operation E7 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. Figures (A) to (D) are explanatory diagrams (part 2) showing an example of operation E7 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 1. This is a block diagram schematically showing the configuration of the learning data collection device according to Embodiment 2. This is a flowchart showing the operation of the learning data collection device according to Embodiment 2. Figures (A) to (E) are explanatory diagrams showing an example of operation E8 of the inference unit, multi-perspective output generation unit, and calculation unit of the learning data collection device according to Embodiment 2.This is a schematic block diagram showing the configuration of the learning data collection device according to Embodiment 3. This is a flowchart showing the operation of the learning data collection device according to Embodiment 3. This is a schematic block diagram showing the configuration of the learning data collection device according to Embodiment 4. This is a flowchart showing the operation of the learning data collection device according to Embodiment 4. This is a schematic block diagram showing the configuration of the learning data collection device according to Embodiment 5.

[0010] The learning data collection device, learning data collection method, and learning data collection program according to the embodiments will be described below with reference to the drawings. The learning data collection device, learning data collection method, and learning data collection program according to the embodiments can facilitate the collection of learning data that can improve the performance of the object detection model. The following embodiments are merely examples, and it is possible to combine embodiments as appropriate and modify each embodiment as appropriate. In the figures, components having the same or similar functions are denoted by the same reference numerals.

[0011] (1) Embodiment 1 (1-1) Configuration Diagram 1 is a block diagram schematically showing the configuration of the learning data collection device 1 according to Embodiment 1. The learning data collection device 1 is a device capable of implementing the learning data collection method according to Embodiment 1. The learning data collection device 1 is a computer as an information processing device capable of executing the learning data collection program according to Embodiment 1. The computer may be a system (i.e., a computer system) including a plurality of information processing devices connected to each other via a network.

[0012] As shown in Figure 1, the learning data collection device 1 includes a learning device 20 that generates a trained model, an inference unit 30 as an inference device (i.e., an object detection unit) that performs inference, a multi-perspective output generation unit 50 that performs inference with a trained model different from that of the inference unit 30, a calculation unit 60, and a collection unit 70. The learning data collection device 1 may also include a storage device for storing information, for example, a first storage unit 10 that stores a training dataset containing a plurality of labeled learning data T10, a second storage unit 40 that stores an image dataset containing a plurality of unlabeled data T40, and a third storage unit 80 that stores collected data C70 collected by the collection unit 70. The first storage unit 10, the second storage unit 40, and the third storage unit 80 do not need to be part of the learning data collection device 1, and may be part of an external device that is communicably connected to the learning data collection device 1. The first storage unit 10, the second storage unit 40, and the third storage unit 80 may each be different storage devices, or they may be different storage areas of a common storage device.

[0013] The learning device 20 has a first learning unit 21. The first learning unit 21 acquires labeled learning data T10, which includes learning images and labels, from a learning dataset, and generates a first object detection model 31 as a trained model by learning using the labeled learning data T10.

[0014] The inference unit 30 acquires unlabeled data T40 containing the target image from the image dataset, detects objects in the target image using the first object detection model 31, and outputs a first detection result D31 as the detection result. For example, the first detection result D31 of the inference unit 30 includes the coordinates of the objects in the target image (for example, the coordinates of the bounding box of a rectangle), the class corresponding to the coordinates, and its likelihood. In other words, the inference unit 30 outputs to the calculation unit 60 the position of the object in the target image and the class of the object, or the position of the object in the target image, the class of the object, and the likelihood of the class. Alternatively, the inference unit 30 may output the coordinates of the object in the target image, the class corresponding to the coordinates, and the likelihood for all classes.

[0015] The multi-perspective output generation unit 50 acquires unlabeled data T40 and generates information about the objects contained in the unlabeled data T40 as the first multi-perspective output D51 using the first multi-perspective model 51, which is a multi-perspective trained model having a different structure from the first object detection model 31.

[0016] In Embodiment 1 (and in other embodiments as well), the object detection model of the inference unit 30 is a trained model that takes an image (still image and moving image) as input and outputs the detected image coordinates, the class corresponding to the image coordinates, and its likelihood. Specific examples of object detection models include the 1-stage object detection models yolo (You Only Look Once) and ssd (Single Shot multibox Detector). Specific examples of object detection models include the 2-stage models R-CNN (Faster Region-based Convolutional Neural Network) and Faster R-CNN (Faster Region-based CNN). Furthermore, specific examples of object detection models include Transformer-type object detection models such as DETR (DETECTION TRANSFORMER) and Swin TRANSFORMER.

[0017] In Embodiment 1 (and similarly in other embodiments), the multi-viewpoint trained model of the multi-viewpoint output generation unit 50 is a trained model having a different structure from the object detection model of the inference unit 30. For example, if the object detection model of the inference unit 30 is a 1-stage model, a 2-stage model or a Transformer model can be used as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50. Also, if the object detection model of the inference unit 30 is a 2-stage model, a 1-stage model or a Transformer model can be used as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50. Also, if the object detection model of the inference unit 30 is a Transformer model, a 1-stage model or a 2-stage model can be used as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50. Furthermore, if the object detection model of the inference unit 30 is a Transformer-type model, a 1-stage model or a 2-stage model can be used as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50. Also, if the detection result by the object detection model of the inference unit 30 includes the coordinates of objects in the target image, the class corresponding to the coordinates, and its likelihood, a natural language model that takes an image and a command as input and outputs natural language can be used as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50.

[0018] The calculation unit 60 calculates a degree of agreement C60, which indicates the degree of agreement between the first detection result D31, which is the detection result, and the first other-viewpoint output D51, which is the output from another viewpoint. The degree of agreement C60 is expressed as a numerical value within a range from 0, which is the lowest degree of agreement, to 1, which is the highest degree of agreement.

[0019] The collection unit 70 collects unlabeled data T40 selected from multiple image datasets based on the degree of agreement C60. For example, the collection unit 70 collects unlabeled data with a degree of agreement C60 lower than a predetermined reference value as collected data C70 and stores the collected data C70 in the third storage unit 80. Alternatively, the collection unit 70 may collect unlabeled data in order from the data with the lowest degree of agreement C60 until the number of data reaches a predetermined number, and store the collected data C70 in the third storage unit 80. The reference value R is set to a number within the range of 0 < R < 1, for example. The image data, which is the collected data C70 stored in the third storage unit 80, has the correct answer added to it and is stored in the first storage unit 10 as labeled image data, updating the training dataset.

[0020] The learning device 20 updates the first object detection model 31 using the training data T10 obtained from the updated training dataset.

[0021] Figure 2 shows an example of the hardware configuration of the training data acquisition device 1. The training data acquisition device 1 includes a processor 101, a memory 102, a storage device 103, and an interface 104. The training data acquisition device 1 is, for example, a computer or a computer system including a plurality of information processing devices. The processor 101 is a CPU (Central Processing Unit), etc. The memory 102 is a volatile semiconductor memory, such as RAM (Random Access Memory). The storage device 103 is a non-volatile storage device, such as a hard disk drive (HDD) or a solid-state drive (SSD). Various types of data, such as labeled training data and unlabeled data, are input to the interface 104.

[0022] Each function of the learning data acquisition device 1 may be implemented by a processing circuit. The processing circuit may be dedicated hardware, or it may be a processor 101 that executes a program stored in memory 102. The processor 101 may be a processing unit, arithmetic unit, microprocessor, microcomputer, or DSP (Digital Signal Processor).

[0023] When the processing circuit is dedicated hardware, the processing circuit may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination of any of these.

[0024] When the processing circuit is a processor 101, the program is implemented by software, firmware, or a combination of software and firmware. The software and firmware are written as a program and stored in memory 102. The processor 101 can realize the functions of the learning data collection device 1 shown in Figure 1 by reading and executing the program stored in memory 102.

[0025] (1-2) Operation diagram 3 is a flowchart showing the operation of the learning data collection device 1 (i.e., the learning data collection method). As shown in Figure 3, the learning data collection method according to Embodiment 1 includes the steps of: acquiring labeled learning data T10 from a learning dataset and generating a first object detection model 31 as a trained model by learning using the labeled learning data T10 (S11); and acquiring unlabeled data T40 including a target image from an image dataset and detecting objects in the target image using the first object detection model 31 and outputting a first detection result D31 (S12). The learning data collection method according to Embodiment 1 also includes the step of acquiring unlabeled data T40 and generating information about objects contained in the unlabeled data T40 as a first other-viewpoint output D51 using a first other-viewpoint model 51 having a different structure from the first object detection model 31 (S13). Furthermore, the learning data collection method according to Embodiment 1 includes the steps of calculating a degree of agreement C60 indicating the degree of agreement between the first detection result D31 and the first other-viewpoint output D51 (S14), and collecting unlabeled data with a degree of agreement C60 lower than a predetermined standard value as collected data C70 (S15, S16). The processing in steps S12 to S16 is performed sequentially for each of the unlabeled data T40 (step S17).

[0026] 《1-3》Operation Example E1 Operation Example E1 shows an example in which a model that performs captioning (the task of outputting natural language text that describes an image) is used as the multi-viewpoint trained model used by the multi-viewpoint output generation unit 50. In this case, the multi-viewpoint output generation unit 50 infers a sentence in natural language and provides it to the calculation unit 60. The calculation unit 60 calculates the degree of agreement C60 between the class inferred in object detection and the class that appeared in the sentence created in natural language. The degree of agreement C60 is "1" if these classes match perfectly, and "0" if there are no matching classes. For example, the degree of agreement C60 is the value obtained by dividing the number of object detections by the total number of classes that appeared in natural language. For example, if a cat is detected in object detection (i.e., the number of matching classes is "1"), and cats and dogs appear in natural language (i.e., the number of appearing classes is "2"), the degree of agreement C60 is calculated by the following formula (1).

[0027]

[0028] Furthermore, if the likelihood of object detection with matching classes is P and the likelihood of natural language is Q, the degree of matching C60a may be calculated using the following equation (3). In equation (3), the sum of likelihood P and likelihood Q (P + Q) is divided by ((number of classes that appeared) × 2), and the resulting value is calculated using the following equation (2).

[0029] As shown in equation (3), the degree of agreement C60a is calculated by adding the value obtained from C60 calculated in equation (1) and the value calculated in equation (2), and then dividing the result by 2. In equation (3), if the maximum value of likelihood P is set to 1 and the maximum value of likelihood Q is set to 1, then the maximum value of the sum (P + Q) is 2, and the maximum value of the degree of agreement C60a is 1. Note that when calculating equation (2), if there are multiple likelihoods P for object detection and multiple likelihoods Q for natural language with matching classes, the sum (P + Q) in equation (2) can be the sum of the multiple likelihoods P for object detection with matching classes and the multiple likelihoods Q for natural language.

[0030]

[0031] Figures 4(A) to 4(D) are explanatory diagrams (part 1) showing an example of operation E1 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 4(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 4(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 4(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "There are dogs and cats in this image." as the first multi-perspective output D51. The first multi-perspective output D51 indicates that the detected class includes dogs and cats. Figure 4(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than the predetermined reference value R.

[0032] Figures 5(A) to 5(D) are explanatory diagrams (part 2) showing an example of operation E1 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 5(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 5(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: Dogs and cats" as the first detection result D31. Figure 5(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "There are dogs and cats in this image." as the first multi-perspective output D51. The first multi-perspective output D51 indicates that the detected class includes dogs and cats. Figure 5(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 1, which is greater than the predetermined reference value R.

[0033] Figures 6(A) to 6(D) are explanatory diagrams (part 3) showing an example of operation E1 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 6(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 6(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: Dogs and cats" as the first detection result D31. Figure 6(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "There is a dog in this image." as the first multi-perspective output D51. The first multi-perspective output D51 indicates that the detected class includes dogs. Figure 6(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than the predetermined reference value R.

[0034] 《1-4》Operation Example E2 Figures 7(A) to (D) are explanatory diagrams (part 1) showing operation example E2 of the inference unit 30, the multi-viewpoint output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 7(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-viewpoint output generation unit 50. Figure 7(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 7(C) shows an example in which the first multi-viewpoint model 51 of the multi-viewpoint output generation unit 50 outputs "Detected class: cat" as the first multi-viewpoint output D51. Figure 7(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 1, which is greater than a predetermined reference value R.

[0035] Figures 8(A) to 8(D) are explanatory diagrams (part 2) showing an example of operation E2 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 8(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 8(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 8(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "There are dogs and cats in this image." as the first multi-perspective output D51. The first multi-perspective output D51 indicates that the detected class includes dogs and cats. Figure 8(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than the predetermined reference value R.

[0036] 《1-5》Operation Example E3 Operation Example E3 shows the use of a Visual Question Answering (VQA) model with a multi-perspective trained model. In this case, a question is asked in natural language, and a "yes" or "no" answer is inferred, and the class information inferred as "yes" is provided to the calculation unit 60. The question asks whether the target class exists or not using a natural language question sentence. The calculation unit 60 performs the same matching score C60 calculation as when using captioning.

[0037] Figures 9(A) to (D) are explanatory diagrams showing an example of operation E3 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 9(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 9(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 9(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "Question Q: Are there cats?" "Answer A: Yes" and "Question Q: Are there dogs?" "Answer A: Yes" as the first multi-perspective outputs D51. The first multi-perspective output D51 indicates that the detected class includes dogs and cats. Figure 9(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than the predetermined reference value R.

[0038] 《1-6》Operation Example E4 Operation Example E4 describes the use of the generating AI model in the multi-viewpoint trained model of the multi-viewpoint output generation unit 50. In this case, the generating AI model asks a question in natural language, "Please tell me the objects present in the image along with their location information," and infers bounding boxes, which are rectangular frames, and their class information even in natural language. In the calculation unit 60, the object detection results and the natural language results become information about the rectangle and its class. Therefore, the product of the bounding box overlap rate (IOU) (Intersection over Union) and the likelihood of the class (i.e., (likelihood of object detection) × (likelihood of natural language)) is calculated, and these are added together to calculate the following total value: Total value = IOU + ((likelihood of object detection) × (likelihood of natural language)). By calculating the likelihood of the target class in natural language in this way, the degree of agreement C60 can be calculated using the Jensen-Shannon divergence (JS) from the likelihood of all classes in object detection and the likelihood of natural language. Note that since JS approaches 0 as the similarity increases, in order to match the scale with IOU, the value obtained by subtracting the product of the likelihood from 1 is used as the degree of agreement C60. The formula for this calculation is shown below. Consistency = IOU + (1 - JS (likelihood of object detection, likelihood of natural language processing))

[0039] Figures 10(A) to (D) are explanatory diagrams showing an example of operation E4 of the inference unit 30, the multi-viewpoint output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 10(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-viewpoint output generation unit 50. Figure 10(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 10(C) shows an example in which the first multi-viewpoint model 51 of the multi-viewpoint output generation unit 50 outputs "In this image, there is a dog at (x1, y1, w1, h1) and a cat at (x2, y2, w2, h2)." as the first multi-viewpoint output D51. Here, x1 and x2 are the x-axis coordinates of the bounding box, y1 and y2 are the y-axis coordinates of the bounding box, w1 and w2 are the width (length) of the bounding box in the x-axis direction, and h1 and h2 are the width (height) of the bounding box in the y-axis direction. Figure 10(D) shows an example where the degree of agreement C60 output from the calculation unit 60 is smaller than the reference value R.

[0040] 《1-7》Operation Example E5 Operation Example E5 shows an example in which the multi-viewpoint trained model of the multi-viewpoint output generation unit 50 calculates the degree of match C60 based on the output obtained from a single prompt of the generating AI (Artificial Intelligence). A prompt refers to a written instruction such as "Please tell me what objects are present in the image." When using the generating AI model for the multi-viewpoint trained model, the AI ​​infers an answer to the question "Please tell me what objects are present in the image" as a natural language question, and provides the calculation unit 60 with class presence information, setting the degree of match C60 to "1" if an object is present in the image, and to "0" otherwise. The calculation unit 60 performs the same calculation of the degree of match C60 as when using captioning.

[0041] Figures 11(A) to (D) are explanatory diagrams showing an example of operation E5 of the inference unit 30, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 11(A), the same unlabeled data T40, including images, is input to the inference unit 30 and the multi-perspective output generation unit 50. Figure 11(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Detected class: cat" as the first detection result D31. Figure 11(C) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "Prompt: Please tell me what objects are present in the image." "Answer: Dogs and cats." as the first multi-perspective output D51. Figure 11(D) shows an example in which the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than the predetermined reference value R. In example E5, by using a different model from the object detection model of the inference unit 30 as the multi-viewpoint trained model of the multi-viewpoint output generation unit 50, the object detection model of the inference unit 30 can be updated using images in which undetected objects are present.

[0042] 《1-8》Operation Example E6 Operation Example E6 shows an example in which the multi-viewpoint output generation unit 50 calculates the degree of agreement C60 based on the output obtained from multiple prompts of the generating AI. When using the generating AI model with a multi-viewpoint trained model, the calculation unit 60 asks questions in natural language that are expected to elicit the same answer to multiple different questions, such as "Please tell me what objects are in the image," and "Are there cats or dogs in the image?", and calculates the degree of agreement C60 based on the results. The calculation unit 60 calculates the degree of agreement based on a total of three results consisting of one object detection result and two results from the multi-viewpoint trained model (for example, the degree of agreement between result #1 and result #2, the degree of agreement between result #2 and result #3, and the degree of agreement between result #3 and result #1), and calculates the average (average of multiple degrees of agreement).

[0043] FIGS. 12(A) to (D) are explanatory diagrams showing an operation example E6 of the inference unit 30, the other viewpoint output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in FIG. 12(A), the same unlabeled data T40 including an image is input to the inference unit 30 and the other viewpoint output generation unit 50. FIG. 12(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "detected class: cat" as the first detection result D31. FIG. 12(C) shows an example in which the first other viewpoint model 51 of the other viewpoint output generation unit 50 outputs "prompt 1: Please tell me the objects existing in the image.", "prompt 2: Are there dogs and cats in the image?", and "answer A: There are dogs and cats" as the first other viewpoint output D51. FIG. 12(D) shows an example in which the degree of coincidence C60 output from the calculation unit 60 is smaller than a predetermined reference value R. In the operation example E6, by using a model different from the object detection model of the inference unit 30 as the other viewpoint learned model of the other viewpoint output generation unit 50, the object detection model of the inference unit 30 can be updated using the image when there is an undetected object in the image. Further, by using the answers obtained for a plurality of prompts of the generation AI, it is possible to reduce the occurrence of undetected (detection omission) due to the ambiguity of the natural language used in the prompts.

[0044] 《1-9》Operation Example E7 The operation example E7 shows an example in which the generation AI generates a natural language question sentence using the object detection result by the inference unit 30, the other viewpoint output generation unit 50 determines whether the answer to the question sentence is correct, and this determination is also output to the calculation unit 60. As the question, when a dog and a cat are detected by the other viewpoint output generation unit 50, if it is found from the information of the detection frame that the dog is on the right side of the cat, a question sentence "Is there a dog on the right side of the cat?" is generated. When the dog and the cat are arranged vertically in the image, a question sentence such as "Is there a dog beside the cat?" or "Are there a dog and a cat?" is generated. In this way, the generation AI asks a question to judge the validity of the detection result and receives the answer as "yes" or "no". The calculation unit 60 calculates the degree of coincidence C60 based on the object detection result and the output of the other viewpoint learned model whose answer is "yes".

[0045] Figures 13(A) to (D) are explanatory diagrams (part 1) showing an operation example E7 of the inference unit 30, the other perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 13(A), the same unlabeled data T40 including an image is input to the inference unit 30 and the other perspective output generation unit 50. Figure 13(B) shows an example in which the first object detection model

[0046] and the first detection result D31 outputs "Prompt 1: Is there a dog on the right side of the cat?" "Answer A: No". Figure 13(C) shows an example in which the first other perspective model 51 of the other perspective output generation unit 50 outputs "There are a dog and a cat in this image." as the first other perspective output D51. Figure 13(D) shows an example in which the degree of coincidence C60 output from the calculation unit 60 is 0.5 and is smaller than a predetermined reference value R.

[0046] Figures 14(A) to (D) are explanatory diagrams (part 2) showing an operation example E7 of the inference unit 30, the other perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 1. As shown in Figure 14(A), the same unlabeled data T40 including an image is input to the inference unit 30 and the other perspective output generation unit 50. Figure 14(B) shows an example in which the first object detection model 31 of the inference unit 30 outputs "Prompt 1: Is there a dog on the right side of the cat?" "Answer A: Yes" as the first detection result D31. Figure 14(C) shows an example in which the first other perspective model 51 of the other perspective output generation unit 50 outputs "There are a dog and a cat in this image." as the first other perspective output D51. Figure 14(D) shows an example in which the degree of coincidence C60 output from the calculation unit 60 is 1 and is larger than a predetermined reference value R. In the operation example E7, by using a model different from the object detection model of the inference unit 30 as the other perspective learned model of the other perspective output generation unit 50, the object detection model of the inference unit 30 can be updated using the image when there is an undetected object in the image. Also, since the prompt of the generation AI can be created using the positional relationship of object detection and the information of the object detection class, it becomes easier to determine the correctness of the answer to the prompt.

[0047]

[0047] By using the learning data collection device 1, learning data collection method, and learning data collection program according to Embodiment 1, image data with a matching score C60 lower than the reference value (i.e., image data that is highly likely to contain undetected objects or false detections in the object detection results by the first object detection model 31) is collected as collected data C70, and the learning dataset is updated using this data, thereby improving the performance of the first object detection model 31.

[0048] By using the learning data collection device 1, learning data collection method, and learning data collection program according to Embodiment 1, annotation, which is the process of adding annotations to objects depicted in an image, can be easily performed. Therefore, the learning data collection device 1, learning data collection method, and learning data collection program according to Embodiment 1 are annotation support, annotation support method, and annotation support program.

[0049] (2) Embodiment 2 Figure 15 is a block diagram schematically showing the configuration of the learning data collection device 2 according to Embodiment 2. The learning data collection device 2 is a device capable of implementing the learning data collection method according to Embodiment 2. The learning data collection device 2 is, for example, a computer capable of executing the learning data collection program according to Embodiment 2.

[0050] As shown in Figure 15, the learning data collection device 2 includes a learning device 20a that generates a trained model, an inference unit 30a that performs inference as an inference device, a multi-perspective output generation unit 50 that performs inference with a multi-perspective trained model different from that of the inference unit 30a, a calculation unit 60, and a collection unit 70.

[0051] The learning device 20a includes a first learning unit 21 and a second learning unit 22. The first learning unit 21 acquires labeled learning data T10 and generates a first object detection model 31 as a trained model by learning using the labeled learning data T10. The second learning unit 22 acquires labeled learning data T10 and generates a second object detection model 32 as a trained model by learning using the labeled learning data T10.

[0052] The inference unit 30a acquires unlabeled data T40 containing the target image from the image dataset, detects objects in the target image using the first object detection model 31 and outputs a first detection result D31 as the detection result, and detects objects in the target image using the second object detection model 32 and outputs a second detection result D32 as the detection result. For example, each of the first detection result D31 and the second detection result D32 of the inference unit 30a includes, for example, the coordinates of the objects in the target image, the class corresponding to the coordinates, and its likelihood. The first object detection model 31 and the second object detection model 32 are trained models with different structures. However, the first object detection model 31 and the second object detection model 32 may be trained models with the same structure.

[0053] The multi-perspective output generation unit 50 acquires unlabeled data T40 and generates information about the objects contained in the unlabeled data T40 as the first multi-perspective output D51 using the first multi-perspective model 51, which is a multi-perspective trained model having a structure different from both the first object detection model 31 and the second object detection model 32. The first multi-perspective model 51 is a pre-prepared model.

[0054] The calculation unit 60 outputs a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-view output D51 (i.e., the first value) and the degree of agreement between the second detection result D32 and the first other-view output D51 (i.e., the second value). This degree of agreement C60 is the average of the first value and the second value (for example, the average of a predetermined number of first values ​​and a predetermined number of second values).

[0055] The collection unit 70 collects unlabeled data T40 selected from multiple image datasets based on the matching score C60. The image data C70, which is the collected data, is assigned a correct answer and stored in the third storage unit 80 as labeled image data, and the training dataset is updated. The assignment of the correct answer is performed by the operator, but it may also be performed automatically based on previously acquired correct answer data.

[0056] The first learning unit 21 of the learning device 20 updates the first object detection model 31 using the learning data T10 obtained from the updated learning dataset. The second learning unit 22 of the learning device 20 updates the second object detection model 32 using the learning data T10 obtained from the updated learning dataset.

[0057] Figure 16 is a flowchart showing the operation of the learning data collection device 2 (i.e., the learning data collection method). As shown in Figure 16, the learning data collection method according to Embodiment 2 includes the steps of: acquiring labeled learning data T10 from a learning dataset and generating a first object detection model 31 and a second object detection model 32 by learning using the labeled learning data T10 (S21); and acquiring unlabeled data T40 including a target image from an image dataset and detecting objects in the target image using the first and second object detection models 31 and 32 and outputting first and second detection results D31 and D32 (S22). Furthermore, the learning data collection method according to Embodiment 2 includes the step of acquiring unlabeled data T40 and generating information about objects contained in the unlabeled data T40 as a first other-viewpoint output D51 using a first other-viewpoint model 51 having a structure different from both the first object detection model 31 and the second object detection model 32 (S23). Furthermore, the learning data collection method according to Embodiment 2 includes the steps of: outputting a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-viewpoint output D51, and the degree of agreement between the second detection result D32 and the first other-viewpoint output D51 (S24); and collecting unlabeled data with a degree of agreement C60 lower than a predetermined standard value as collected data C70 (S25, S26). The processing in steps S22 to S26 is performed sequentially for each of the unlabeled data T40 (step S27).

[0058] Figures 17(A) to (E) are explanatory diagrams showing an example of operation E8 of the inference unit 30a, the multi-perspective output generation unit 50, and the calculation unit 60 of the learning data collection device 2. As shown in Figure 17(A), the same unlabeled data T40, including images, is input to the inference unit 30a and the multi-perspective output generation unit 50. Figure 17(B) shows an example in which the first object detection model 31 of the inference unit 30a outputs "Detected class: cat" as the first detection result D31. Figure 17(C) shows an example in which the second object detection model 32 of the inference unit 30a outputs "Detected class: cat" as the second detection result D32. Figure 17(D) shows an example in which the first multi-perspective model 51 of the multi-perspective output generation unit 50 outputs "There are dogs and cats in this image." as the first multi-perspective output D51. The first multi-perspective output D51 indicates that the detected class includes dogs and cats. Figure 17(E) shows an example where the degree of agreement C60 output from the calculation unit 60 is 0.5, which is smaller than a predetermined reference value R (for example, R = 0.8).

[0059] By using the learning data collection device 2, learning data collection method, and learning data collection program according to Embodiment 2, image data with a matching score C60 lower than the reference value (i.e., image data in which the object detection results by the first and second object detection models 31 and 32 are likely to include undetected objects or false detections) are collected as collected data C70, and the learning dataset is updated using this data, thereby improving the performance of the first and second object detection models 31 and 32.

[0060] With respect to matters other than those described above, Embodiment 2 is the same as Embodiment 1.

[0061] (3) Embodiment 3 Figure 18 is a block diagram schematically showing the configuration of the learning data collection device 3 according to Embodiment 3. The learning data collection device 3 is a device capable of implementing the learning data collection method according to Embodiment 3. The learning data collection device 3 is, for example, a computer capable of executing the learning data collection program according to Embodiment 3.

[0062] As shown in Figure 18, the learning data collection device 3 includes a learning device 20 that generates a trained model, an inference unit 30 that performs inference as an inference device, a multi-perspective output generation unit 50a that performs inference with a trained model different from that of the inference unit 30, a calculation unit 60, and a collection unit 70.

[0063] The learning device 20 has a first learning unit 21. The first learning unit 21 acquires labeled learning data T10 and generates a first object detection model 31 as a trained model by learning using the labeled learning data T10.

[0064] The inference unit 30 acquires unlabeled data T40 containing the target image from the image dataset, detects objects in the target image using the first object detection model 31, and outputs a first detection result D31 as the detection result.

[0065] The multi-perspective output generation unit 50a acquires unlabeled data T40, generates information about the objects contained in the unlabeled data T40 as the first multi-perspective output D51 using the first multi-perspective model 51 which has a different structure from the first object detection model 31, and generates information about the objects contained in the unlabeled data T40 as the first multi-perspective output D52 using the second multi-perspective model 52 which has a different structure from the first object detection model 31. The first multi-perspective model 51 and the second multi-perspective model 52 are pre-prepared models. The first multi-perspective model 51 and the second multi-perspective model 52 are trained models with different structures. However, the first multi-perspective model 51 and the second multi-perspective model 52 may be trained models with the same structure.

[0066] The calculation unit 60 outputs a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-view output D51 (i.e., the third value) and the degree of agreement between the first detection result D31 and the second other-view output D52 (i.e., the fourth value). This degree of agreement C60 is, for example, the average of the third value and the fourth value.

[0067] The collection unit 70 collects unlabeled data T40 selected from multiple image datasets based on the degree of agreement C60. The image data C70, which is the collected data, is stored in the third storage unit 80, and the correct answer is added to it. This data is then stored in the first storage unit 10 as labeled image data, and the training dataset is updated.

[0068] The first learning unit 21 of the learning device 20 updates the first object detection model 31 using the learning data T10 obtained from the updated learning dataset.

[0069] Figure 19 is a flowchart showing the operation of the learning data collection device 3 (i.e., the learning data collection method). As shown in Figure 19, the learning data collection method according to Embodiment 3 includes the steps of: acquiring labeled learning data T10 from a learning dataset and generating a first object detection model 31 by learning using the labeled learning data T10 (S31); and acquiring unlabeled data T40 including a target image from an image dataset and detecting objects in the target image using the first object detection model 31 and outputting a first detection result D31 (S32). Furthermore, the learning data collection method according to Embodiment 3 includes the steps of: acquiring unlabeled data T40, generating information about objects contained in the unlabeled data T40 as a first other-viewpoint output D51 using a first other-viewpoint model 51 having a different structure from the first object detection model 31, and generating information about objects contained in the unlabeled data T40 as a second other-viewpoint output D52 using a second other-viewpoint model 52 having a different structure from the first object detection model 31 (S33). Furthermore, the learning data collection method according to Embodiment 3 includes the steps of: outputting a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-viewpoint output D51 and the degree of agreement between the first detection result D31 and the second other-viewpoint output D52 (S34); and collecting unlabeled data with a degree of agreement C60 lower than a predetermined standard value as collected data C70 (S35, S36). The processing in steps S32 to S36 is performed sequentially for each of the unlabeled data T40 (step S37).

[0070] By using the learning data collection device 3, learning data collection method, and learning data collection program according to Embodiment 3, image data with a matching score C60 lower than the reference value (i.e., image data in which there is a high possibility that the object detection result by the first object detection model 31 includes undetected objects or false detections) is collected as collected data C70, and the learning dataset is updated using this data, thereby improving the performance of the first object detection model 31.

[0071] With respect to matters other than those described above, Embodiment 3 is the same as Embodiment 1.

[0072] (4) Embodiment 4 Figure 20 is a block diagram schematically showing the configuration of the learning data collection device 4 according to Embodiment 4. The learning data collection device 4 is a device capable of implementing the learning data collection method according to Embodiment 4. The learning data collection device 4 is a computer capable of executing the learning data collection program according to Embodiment 4.

[0073] As shown in Figure 20, the learning data collection device 4 includes a learning device 20a that generates a trained model, an inference unit 30a that performs inference as an inference device, a multi-perspective output generation unit 50a that performs inference with a multi-perspective trained model different from that of the inference unit 30a, a calculation unit 60, and a collection unit 70.

[0074] The learning device 20a and the inference unit 30a are the same as those described in Embodiment 2. The other-viewpoint output generation unit 50a is the same as that described in Embodiment 3.

[0075] The calculation unit 60 outputs a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-view output D51 (i.e., the fifth value), the degree of agreement between the second detection result D32 and the first other-view output D51 (i.e., the sixth value), the degree of agreement between the first detection result D31 and the second other-view output D52 (i.e., the seventh value), and the degree of agreement between the second detection result D32 and the second other-view output D52 (i.e., the eighth value). This degree of agreement C60 is the average value of the fifth value, the sixth value, the seventh value, and the eighth value (for example, the average value of a predetermined number of fifth values, a predetermined number of sixth values, a predetermined number of seventh values, and a predetermined number of eighth values).

[0076] The collection unit 70 collects unlabeled data T40 selected from multiple image datasets based on the matching score C60. The image data C70, which is the collected data, is assigned a correct answer and stored in the third storage unit 80 as labeled image data, and the training dataset is updated. The assignment of the correct answer is performed by the operator, but it may also be performed automatically based on previously acquired correct answer data.

[0077] The first learning unit 21 of the learning device 20 updates the first object detection model 31 using the learning data T10 obtained from the updated learning dataset. The second learning unit 22 of the learning device 20 updates the second object detection model 32 using the learning data T10 obtained from the updated learning dataset.

[0078] Figure 21 is a flowchart showing the operation of the learning data collection device 4 (i.e., the learning data collection method). As shown in Figure 21, the learning data collection method according to Embodiment 4 includes the steps of: acquiring labeled learning data T10 from a learning dataset and generating a first object detection model 31 and a second object detection model 32 by learning using the labeled learning data T10 (S41); and acquiring unlabeled data T40 including a target image from an image dataset and detecting objects in the target image using the first and second object detection models 31 and 32 and outputting first and second detection results D31 and D32 (S42). Furthermore, the learning data collection method according to Embodiment 4 includes the steps (S43) of acquiring unlabeled data T40, generating information about objects contained in the unlabeled data T40 as a first other-viewpoint output D51 using a first other-viewpoint model 51 having a structure different from both the first and second object detection models 31 and 32, and generating information about objects contained in the unlabeled data T40 as a second other-viewpoint output D52 using a second other-viewpoint model 52 having a structure different from both the first and second object detection models 31 and 32. Furthermore, the learning data collection method according to Embodiment 4 includes the steps of: outputting a degree of agreement C60 calculated based on the degree of agreement between the first detection result D31 and the first other-viewpoint output D51, the degree of agreement between the second detection result D32 and the first other-viewpoint output D51, the degree of agreement between the first detection result D31 and the second other-viewpoint output D52, and the degree of agreement between the second detection result D32 and the second other-viewpoint output D52 (S44); and collecting unlabeled data with a degree of agreement C60 lower than a predetermined standard value as collected data C70 (S45, S46). The processing in steps S42 to S46 is performed sequentially for each of the unlabeled data T40 (step S47).

[0079] By using the learning data collection device 4, learning data collection method, and learning data collection program according to Embodiment 4, image data with a matching score C60 lower than the reference value (i.e., image data in which the object detection results by the first and second object detection models 31 and 32 are likely to include undetected objects or false detections) are collected as collected data C70, and the learning dataset is updated using this data, thereby improving the performance of the first and second object detection models 31 and 32.

[0080] With respect to matters other than those described above, Embodiment 4 is the same as any of Embodiments 1 to 3.

[0081] (5) Embodiment 5 Figure 22 is a block diagram schematically showing the configuration of the learning data collection device 5 according to Embodiment 5. The learning data collection device 5 shown in Figure 22 differs from the learning data collection device 1 shown in Figure 1 in that it has a learning data creation unit 81. The learning data creation unit 81 adds correct answer data to the collected data C70 (image data) to create labeled learning data and adds it to the learning dataset. Furthermore, the hardware configuration of the learning data collection device 5 according to Embodiment 5 is the same as that described in Embodiment 1. With respect to all other matters, Embodiment 5 is the same as Embodiment 1.

[0082] By using the learning data collection device 5, learning data collection method, and learning data collection program according to Embodiment 5, image data with a matching score C60 lower than a predetermined reference value (i.e., image data in which the object detection results by the first and second object detection models 31 and 32 are likely to include undetected objects or false detections) is collected as collected data C70, and the learning dataset is updated using this data, thereby improving the performance of the first object detection model 31.

[0083] Furthermore, the learning data creation unit 81 can be applied to any of the learning data collection devices 2 to 4 according to Embodiments 2 to 4.

[0084] (6) Various aspects of the present disclosure Various aspects of the present disclosure are described below as appendices. (Appendix 1) A learning device comprising: a learning device that acquires labeled learning data including learning images and labels from a learning dataset and generates a trained model by learning using the labeled learning data; an inference unit that acquires unlabeled data including a target image from an image dataset, detects objects in the target image using the trained model and outputs a detection result; a multi-view output generation unit that acquires the unlabeled data and generates information about objects contained in the unlabeled data as multi-view output using a multi-view trained model having a different structure from the trained model; a calculation unit that calculates a degree of agreement indicating the degree of agreement between the detection result and the multi-view output; and a collection unit that collects the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data. (Appendix 2) The learning device according to Appendix 1, wherein the learning device updates the trained model using the learning dataset including the collected data. (Note 3) The learning data collection device according to Note 1 or 2, characterized in that the detection result of the inference unit includes the coordinates of the object shown in the target image and the likelihood of the class corresponding to the coordinates. (Note 4) The learning data collection device according to any one of Notes 1 to 3, characterized in that the trained model includes a first object detection model that detects the object shown in the target image and outputs a first detection result, the detection result includes a first detection result generated using the first object detection model, the other-view trained model includes a first other-view model that detects the object shown in the target image and outputs a first other-view output, the other-view output includes a first other-view output generated using the first other-view model, and the calculation unit outputs the degree of agreement between the first detection result and the first other-view output as the degree of agreement.(Note 5) The training model includes a first object detection model that detects the object in the target image and outputs a first detection result, and a second object detection model that detects the object in the target image and outputs a second detection result, the detection result includes a first detection result generated using the first object detection model and a second detection result generated using the second object detection model, the multi-viewpoint training model includes a first multi-viewpoint model that detects the object in the target image and generates a first multi-viewpoint output, the multi-viewpoint output includes a first multi-viewpoint output generated using the first multi-viewpoint model, and the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first multi-viewpoint output and the degree of agreement between the second detection result and the first multi-viewpoint output, the training data collection device according to any one of Notes 1 to 3. (Note 6) The training data collection device according to any one of Notes 1 to 3, characterized in that the trained model includes a first object detection model that detects the object in the target image and outputs a first detection result, the detection result includes a first detection result generated using the first object detection model, the other-view trained model includes a first other-view model that detects the object in the target image and generates a first other-view output, and a second other-view model that detects the object in the target image and generates a second other-view output, the other-view output includes a first other-view output generated using the first other-view model and a second other-view output generated using the second other-view model, and the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first other-view output and the degree of agreement between the first detection result and the second other-view output.(Note 7) The trained model includes a first object detection model that detects the object in the target image and outputs a first detection result, and a second object detection model that detects the object in the target image and outputs a second detection result, the detection result includes a first detection result generated using the first object detection model and a second detection result generated using the second object detection model, the multi-view trained model includes a first multi-view model that detects the object in the target image and generates a first multi-view output, and a second multi-view model that detects the object in the target image and generates a second multi-view output, the multi-view output includes a first multi-view output generated using the first multi-view model and a second multi-view output generated using the second multi-view model, The learning data collection device according to any one of Appendix 1 to 3, characterized in that the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first other-viewpoint output, the degree of agreement between the first detection result and the second other-viewpoint output, the degree of agreement between the second detection result and the first other-viewpoint output, and the degree of agreement between the second detection result and the second other-viewpoint output. (Appendix 8) The learning data collection device according to any one of Appendix 1 to 7, further comprising a learning data creation unit that adds the collected data to the learning dataset. (Appendix 9) The learning data collection device according to any one of Appendix 1 to 8, characterized in that the other-viewpoint trained model of the other-viewpoint output generation unit includes a natural language model. (Appendix 10) The learning data collection device according to any one of Appendix 1 to 9, characterized in that the other-viewpoint output generation unit outputs to the calculation unit the classes for which it has received a "yes" answer to a natural language question about whether or not a target class exists. (Note 11) The learning data collection device according to any one of Notes 1 to 8, characterized in that the other viewpoint output generation unit outputs to the calculation unit the position of the object and the class of the object as it appears in the target image, or the position of the object, the class of the object, and the likelihood of the class as it appears in the target image.(Note 12) The learning data collection device according to any one of Notes 1 to 8, characterized in that the multi-perspective output generation unit outputs information obtained from a single prompt of the generating AI to the calculation unit. (Note 13) The learning data collection device according to any one of Notes 1 to 8, characterized in that the multi-perspective output generation unit outputs information obtained from multiple prompts of the generating AI to the calculation unit. (Note 14) The learning data collection device according to any one of Notes 1 to 8, characterized in that the multi-perspective output generation unit outputs a determination to the calculation unit whether the answer to the question is correct in response to a question about whether a target class exists. (Note 15) The learning data collection device according to any one of Notes 1 to 8, characterized in that the multi-perspective output generation unit outputs classes present in the image, and the calculation unit calculates the degree of agreement between the classes detected by the inference unit and the classes detected by the multi-perspective trained model. (Note 16) A method for collecting learning data that is performed by a learning data collection device, comprising: a step of acquiring labeled learning data including a learning image and a label from a learning dataset, and generating a trained model by training using the labeled learning data; a step of acquiring unlabeled data including a target image from an image dataset, and detecting an object in the target image using the trained model and outputting a detection result; a step of acquiring the unlabeled data, and generating information about the object contained in the unlabeled data as an output from another viewpoint using a trained model from another viewpoint having a different structure from the trained model; a step of calculating a degree of agreement that indicates the degree of agreement between the detection result and the output from another viewpoint; and a step of collecting the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.(Note 17) A learning data collection program characterized by causing a computer to perform the following steps: acquire labeled learning data including learning images and labels from a learning dataset, and generate a trained model by learning using the labeled learning data; acquire unlabeled data including a target image from an image dataset, and use the trained model to detect objects in the target image and output a detection result; acquire the unlabeled data, and use a multi-view trained model having a different structure from the trained model to generate information about objects contained in the unlabeled data as multi-view output; calculate a degree of agreement indicating the degree of agreement between the detection result and the multi-view output; and collect the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.

[0085] 1-5 Learning data collection device, 10 First storage unit, 20, 20a Learning device, 21 First learning unit, 22 Second learning unit, 30, 30a Inference unit (object detection unit), 31 First object detection model (trained model), 32 Second object detection model (trained model), 40 Second storage unit, 50, 50a Multi-perspective output generation unit, 51 First multi-perspective model (multi-perspective trained model), 52 Second multi-perspective model (multi-perspective trained model), 60 Calculation unit, 70 Collection unit, 80 Third storage unit, T10 Labeled learning data, D31, D32 Detection results, T40 Unlabeled data, D51, D52 Multi-perspective output, C60 Matching score, C70 Collected data.

Claims

1. A learning device comprising: a learning device that acquires labeled learning data including learning images and labels from a learning dataset and generates a trained model by learning using the labeled learning data; an inference unit that acquires unlabeled data including a target image from an image dataset, detects objects in the target image using the trained model and outputs a detection result; a multi-view output generation unit that acquires the unlabeled data and generates information about objects contained in the unlabeled data as multi-view output using a multi-view trained model having a different structure from the trained model; a calculation unit that calculates a degree of agreement indicating the degree of agreement between the detection result and the multi-view output; and a collection unit that collects the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.

2. The learning data collection device according to claim 1, characterized in that the learning device updates the trained model using the learning dataset which includes the collected data.

3. The learning data collection device according to claim 1 or 2, characterized in that the detection result of the inference unit includes the coordinates of the object shown in the target image and the likelihood of the class corresponding to the coordinates.

4. The training data collection device according to any one of claims 1 to 3, characterized in that the trained model includes a first object detection model that detects the object in the target image and outputs a first detection result, the detection result includes a first detection result generated using the first object detection model, the other-view trained model includes a first other-view model that detects the object in the target image and outputs a first other-view output, the other-view output includes a first other-view output generated using the first other-view model, and the calculation unit outputs the degree of agreement between the first detection result and the first other-view output as the degree of agreement.

5. The training model includes a first object detection model that detects the object in the target image and outputs a first detection result, and a second object detection model that detects the object in the target image and outputs a second detection result, wherein the detection result includes a first detection result generated using the first object detection model and a second detection result generated using the second object detection model, the multi-viewpoint trained model includes a first multi-viewpoint model that detects the object in the target image and generates a first multi-viewpoint output, the multi-viewpoint output includes a first multi-viewpoint output generated using the first multi-viewpoint model, and the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first multi-viewpoint output and the degree of agreement between the second detection result and the first multi-viewpoint output, characterized in that the training data collection device according to any one of claims 1 to 3.

6. The training data collection device according to any one of claims 1 to 3, wherein the trained model includes a first object detection model that detects the object in the target image and outputs a first detection result, the detection result includes a first detection result generated using the first object detection model, the multi-view trained model includes a first multi-view model that detects the object in the target image and generates a first multi-view output, and a second multi-view model that detects the object in the target image and generates a second multi-view output, the multi-view output includes a first multi-view output generated using the first multi-view model and a second multi-view output generated using the second multi-view model, and the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first multi-view output and the degree of agreement between the first detection result and the second multi-view output.

7. The trained model includes a first object detection model that detects the object in the target image and outputs a first detection result, and a second object detection model that detects the object in the target image and outputs a second detection result, the detection result includes a first detection result generated using the first object detection model and a second detection result generated using the second object detection model, the multi-view trained model includes a first multi-view model that detects the object in the target image and generates a first multi-view output, and a second multi-view model that detects the object in the target image and generates a second multi-view output, the multi-view output includes a first multi-view output generated using the first multi-view model and a second multi-view output generated using the second multi-view model, The learning data collection device according to any one of claims 1 to 3, characterized in that the calculation unit outputs the degree of agreement calculated based on the degree of agreement between the first detection result and the first other-viewpoint output, the degree of agreement between the first detection result and the second other-viewpoint output, the degree of agreement between the second detection result and the first other-viewpoint output, and the degree of agreement between the second detection result and the second other-viewpoint output.

8. A learning data collection device according to any one of claims 1 to 7, further comprising a learning data creation unit that adds the collected data to the learning dataset.

9. The learning data collection device according to any one of claims 1 to 8, characterized in that the multi-perspective trained model in the multi-perspective output generation unit includes a natural language model.

10. The learning data collection device according to any one of claims 1 to 9, characterized in that the other-perspective output generation unit outputs to the calculation unit classes for which it has received a "yes" response to a natural language question about whether or not a target class exists.

11. The learning data collection device according to any one of claims 1 to 8, characterized in that the other viewpoint output generation unit outputs to the calculation unit the position of the object and the class of the object as it appears in the target image, or the position of the object and the class of the object as it appears in the target image and the likelihood of the class.

12. The learning data collection device according to any one of claims 1 to 8, characterized in that the other-viewpoint output generation unit outputs information obtained from a single prompt of the generating AI to the calculation unit.

13. The learning data collection device according to any one of claims 1 to 8, characterized in that the other-viewpoint output generation unit outputs information obtained from multiple prompts of the generating AI to the calculation unit.

14. The learning data collection device according to any one of claims 1 to 8, characterized in that the other-perspective output generation unit outputs to the calculation unit a determination of whether the answer to the question is correct in response to the question of whether the target class exists.

15. The learning data collection device according to any one of claims 1 to 8, characterized in that the other-viewpoint output generation unit outputs classes present in the image, and the calculation unit calculates the degree of agreement between the classes detected by the inference unit and the classes detected by the other-viewpoint trained model.

16. A method for collecting learning data, performed by a learning data collection device, comprising: a step of acquiring labeled learning data including learning images and labels from a learning dataset, and generating a trained model by training using the labeled learning data; a step of acquiring unlabeled data including a target image from an image dataset, and detecting objects in the target image using the trained model and outputting a detection result; a step of acquiring the unlabeled data, and generating information about objects contained in the unlabeled data as an output from another viewpoint using a multi-viewpoint trained model having a different structure from the trained model; a step of calculating a degree of agreement indicating the degree of agreement between the detection result and the multi-viewpoint output; and a step of collecting the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.

17. A training data collection program characterized by causing a computer to perform the following steps: acquire labeled training data including training images and labels from a training dataset, and generate a trained model by training using the labeled training data; acquire unlabeled data including a target image from an image dataset, detect objects in the target image using the trained model, and output a detection result; acquire the unlabeled data, and generate information about the objects contained in the unlabeled data as an output from another viewpoint using a multi-viewpoint trained model having a different structure from the trained model; calculate a degree of agreement indicating the degree of agreement between the detection result and the multi-viewpoint output; and collect the unlabeled data whose degree of agreement is lower than a predetermined standard value as collected data.

Images