Learning device, learning method, and learning program

Abstract

To provide a learning device, a learning method, and a learning program, which can efficiently generate teacher data in a weakly supervised learning model.SOLUTION: A learning device 1 comprises an input section 21, a data processing section 23, and a learning section 24. The input section 21 stores image data or moving image data collected from the outside in a teacher data candidate database. The data processing section 23 generates teacher data from the image data or the moving image data acquired from the teacher data candidate database and stores the generated teacher data in the teacher data database. The learning section 24 generates a teacher data file for learning from the data acquired from the teacher data database, generates a learned model by performing weakly supervised learning on the basis of the teacher data file for learning, and stores the generated learned model in the learned model database.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The embodiments relate to a learning device, a learning method, and a learning program. [Background technology]

[0002] Image recognition AI (Artificial Intelligence) is known that can recognize and distinguish specific shapes contained in input images. In order to improve the accuracy of image recognition AI, learning using training data, for example, tens of thousands of images or more, is required. A method for creating training data for image recognition AI is known in which instruction, called annotation, is given pixel by pixel to image data that is a candidate for training data. The task of teaching a huge amount of image data pixel by pixel is time-consuming and costly.

[0003] Furthermore, a weakly supervised learning model is known as a type of image recognition AI. A weakly supervised learning model can learn from training data that teaches only the presence or absence of a shape to be detected on an image-by-image basis. In an image recognition AI to which a weakly supervised learning model is applied, annotation is performed on an image-by-image basis, rather than on a pixel-by-pixel basis. For this reason, annotation work for creating training data used in a weakly supervised learning model can be performed in less time and at less cost than when pixel-by-pixel teaching work is performed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2023-032702 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there are very few annotation tools specialized for image recognition AI that applies weakly supervised learning, and there is an issue that the feature of weakly supervised learning, which can reduce the amount of work required to prepare training data, is not being utilized.

[0006] The present invention has been made to solve the above problems, and aims to provide a learning device, a learning method, and a learning program that are capable of efficiently creating teacher data in a weakly supervised learning model. [Means for solving the problem]

[0007] The learning device of the embodiment includes an input unit, a data processing unit, and a learning unit. The input unit stores image data or video data collected from the outside in a teacher data candidate database. The data processing unit generates teacher data from image data or video data acquired from the teacher data candidate database, and stores the generated teacher data in the teacher data database. The learning unit generates a training teacher data file from data acquired from the teacher data database, generates a trained model by performing weakly supervised learning based on the training teacher data file, and stores the generated trained model in the trained model database. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a block diagram showing an example of the hardware configuration of the learning device according to the first embodiment. [Diagram 2] FIG. 2 is a block diagram showing an example of the functional configuration of the learning device according to the first embodiment. [Diagram 3] FIG. 2 is a block diagram showing an example of the configuration of a data storage unit included in the learning device according to the first embodiment. [Figure 4] 4 is a data table showing an example of the configuration of a teacher data candidate DB stored in a data storage unit included in the learning device according to the first embodiment. [Diagram 5] 4 is a data table showing an example of the configuration of a teacher data DB stored in a data storage unit included in the learning device according to the first embodiment. [Figure 6] 4 is a data table showing an example of the configuration of a learning teacher data file stored in a data storage unit included in the learning device according to the first embodiment. [Figure 7] 4 is a data table showing an example of the configuration of a test data DB stored in a data storage unit included in the learning device according to the first embodiment. [Figure 8] 4 is a data table showing an example of the configuration of an inference result DB stored in a data storage unit included in the learning device according to the first embodiment. [Figure 9] FIG. 2 is a block diagram showing an example of the configuration of a model storage unit included in the learning device according to the first embodiment. [Figure 10] 1 is a data table showing an example of a configuration of a trained model database stored in a model storage unit included in the learning device according to the first embodiment. [Figure 11] 5 is a flowchart showing an example of the operation of an input unit included in the learning device according to the first embodiment. [Figure 12] 5 is a flowchart showing an example of a first operation of a data processing unit included in the learning device according to the first embodiment. [Figure 13] 11 is a flowchart showing an example of a second operation of the data processing unit included in the learning device according to the first embodiment. [Figure 14] 5 is a schematic diagram showing a specific example of a teacher data sorting process performed by a data processing unit included in the learning device according to the first embodiment. FIG. [Figure 15] 1 is a schematic diagram showing a specific example of a GUI of an annotation tool displayed on a display device provided in the learning device according to the first embodiment. [Figure 16] 5 is a flowchart showing an example of the operation of a learning unit included in the learning device according to the first embodiment. [Figure 17] 4 is a schematic diagram showing a specific example of a process of sorting training teacher data by a learning unit included in the learning device according to the first embodiment. FIG. [Figure 18] 5 is a flowchart showing a specific example of the operation of an inference unit included in the learning device according to the first embodiment. [Figure 19] FIG. 2 is a schematic diagram showing an example of an inference result generated by an inference unit included in the learning device according to the first embodiment. [Figure 20] 10 is a flowchart showing a modified example of the operation of the inference unit included in the learning device according to the first embodiment. [Figure 21] 10 is a flowchart showing an example of a second operation of the data processing unit included in the learning device according to the second embodiment. [Figure 22] 13 is a data table showing an example of the configuration of a teacher data DB stored in a data storage unit included in the learning device according to the second embodiment. [Diagram 23] FIG. 11 is a schematic diagram showing a specific example of a GUI of an annotation tool displayed on a display device provided in a learning device according to a second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Each embodiment will be described below with reference to the drawings. Each embodiment illustrates an apparatus or method for embodying the technical idea of ​​the invention. The drawings referred to below are schematic or conceptual. Components having substantially the same functions and configurations are given the same reference numerals. In this specification, "database" will be abbreviated to "DB" as appropriate.

[0010] <1> First embodiment The first embodiment relates to a learning device 1 capable of generating a weakly supervised learning model by using image data stored in a folder associated with tag information. Details of the learning device 1 according to the first embodiment will be described below. Note that in this specification, an example will be given of an image recognition AI using weakly supervised learning being used to detect abnormalities in roads.

[0011] <1-1> Configuration First, the configuration of the learning device 1 according to the first embodiment will be described.

[0012] <1-1-1> Hardware configuration of learning device 1 Fig. 1 is a block diagram showing an example of a hardware configuration of a learning device 1 according to the first embodiment. As shown in Fig. 1, the learning device 1 includes, for example, a processor 11, a memory device 12, a storage device 13, a display device 14, an input interface 15, and a communication module 16.

[0013] The processor 11 is an integrated circuit capable of executing various programs, and controls the overall operation of the learning device 1. As the processor 11, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (field-programmable gate array), a GPU (Graphics Processing Unit), etc. are used.

[0014] The memory device 12 is a storage device that stores programs and control data for controlling the learning device 1. The memory device 12 is used as a work area for the processor 11. Note that the learning device 1 may be provided with a memory device that stores the programs and control data and a memory device that is used as a work area for the processor 11 separately.

[0015] The storage device 13 is a non-volatile storage device. The storage device 13 stores, for example, application software and system software of the learning device 1. As the storage device 13, for example, a solid state drive (SSD) or a hard disk drive (HDD) is used.

[0016] The display device 14 is a device configured to be capable of displaying characters, images, etc. The display device 14 displays, for example, a GUI (Graphical User Interface) corresponding to application software, etc., based on an instruction from the processor 11. As the display device 14, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display, etc., are used.

[0017] Input interface 15 is an interface used to operate study device 1. A user can operate study device 1 using input interface 15. Note that in study device 1, the function of input interface 15 may be built into display device 14.

[0018] The communication module 16 is a circuit used for transmitting and receiving data, etc., with an external device. Either wired communication or wireless communication may be used for transmitting and receiving data, etc., by the communication module 16. The communication module 16 may be connected to an external storage device, or may be connected to an external server via a network.

[0019] Note that learning device 1 may have other hardware configurations. For example, when learning device 1 is an IoT (Internet of Things) terminal, display device 14 may be omitted from learning device 1. Each of storage device 13, display device 14, and input interface 15 may be externally connected to learning device 1.

[0020] <1-1-2> Functional configuration of learning device 1 Fig. 2 is a block diagram showing an example of a functional configuration of the learning device 1 according to the first embodiment. As shown in Fig. 2, the learning device 1 functions as a computer including, for example, an input unit 21, a data storage unit 22, a data processing unit 23, a learning unit 24, a model storage unit 25, and an inference unit 26. Each of the input unit 21, the data storage unit 22, the data processing unit 23, the learning unit 24, the model storage unit 25, and the inference unit 26 can be realized by the hardware configuration of the learning device 1 described using Fig. 1.

[0021] The input unit 21 has a data input function of storing data DAT such as moving images and images input from the outside in a predetermined database of the data storage unit 22. Images and videos from an externally connected camera may be input to the input unit 21 as data DAT.

[0022] The data storage unit 22 stores a plurality of databases. The data storage unit 22 is configured to be accessible by each of the input unit 21, the data processing unit 23, the learning unit 24, and the inference unit 26. The plurality of databases stored in the data storage unit 22 will be described in detail later.

[0023] The data processing unit 23 has a data confirmation function and a teacher data creation function. The data confirmation function extracts data suitable for teacher data from data that is the source of teacher data, such as videos and images stored in the data storage unit 22 and inference result images output by the inference unit 26. The teacher data creation function outputs teacher data after adding information required as teacher data to the image extracted by the data confirmation function and stores it in the data storage unit 22. When creating teacher data from an inference result image, the teacher data creation function may have a function of acquiring an inference result image and an original image from the data storage unit 22, displaying both images simultaneously, and facilitating the specification of the image cut-out range. In addition, the teacher data creation function may create a folder name based on the taught tag information and store the created teacher data in the folder. As a result, the learning unit 24 may create teacher data to be used for learning by associating tag information on an image-by-image basis without adding detailed teaching information on an image-by-image basis.

[0024] The learning unit 24 has a learning preparation function and a weakly supervised learning function. The learning preparation function can extract an image to be used as teacher data (hereinafter, also referred to as a "teacher image") from the data storage unit 22 according to the characteristics of a learning model to be created by the weakly supervised learning function. The learning preparation function selects appropriate teacher data from the data storage unit 22, creates a learning teacher data file, and inputs it to the weakly supervised learning function. The weakly supervised learning function is premised on image recognition AI. The weakly supervised learning function performs weakly supervised learning based on the learning teacher data file and specified parameters provided by the learning preparation function, and stores the generated model in the model storage unit 25. The weakly supervised learning function can generate a learning model based on the presence or absence of a shape to be detected, without requiring teaching information for each teacher image or each pixel.

[0025] The model storage unit 25 stores the trained model created by the learning unit 24. The model storage unit 25 can store a plurality of trained models.

[0026] The inference unit 26 has an inference processing function that performs inference on videos and images or camera-captured images stored in the data storage unit 22 using a trained model stored in the model storage unit 25. The inference unit 26 stores inference results obtained by the inference processing function in the data storage unit 22.

[0027] <1-1-3> Configuration of data storage unit 22 Fig. 3 is a block diagram showing an example of the configuration of the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 3, the data storage unit 22 stores, for example, a teacher data candidate DB221, a teacher data DB222, a training teacher data file 223, a test data DB224, and an inference result DB225.

[0028] The teacher data candidate DB 221 stores data that is a candidate for teacher data, such as videos and images. The data stored in the teacher data candidate DB 221 corresponds to data input by the data input function of the input unit 21. The teacher data creation function of the data processing unit 23 creates teacher data based on appropriate data acquired from the teacher data candidate DB 221 by the data confirmation function of the data processing unit 23.

[0029] The teacher data DB 222 stores images that can be used as teacher data and information about the teacher data in pairs. The teacher data DB 222 has a configuration in which images associated with tag information are stored in a folder to which a name corresponding to the tag information is added.

[0030] The training teacher data file 223 is a group of files in which data extracted from the teacher data DB 222 according to the characteristics of a model that the user wants to create is compiled for training purposes. That is, the training teacher data file 223 stores images to be provided as teacher data to the weakly supervised learning function of the learning unit 24.

[0031] The test data DB 224 stores data that is the target of inference using the trained model, such as videos, images, or images captured by a camera. The data stored in the test data DB 224 corresponds to data input by the data input function of the input unit 21.

[0032] The inference result DB 225 stores an inference result image and an inference score output by the inference processing function of the inference unit 26. The inference result image is, for example, an image showing the position of a defect detected by inference with respect to an original image. The inference score is a numerical value representing the similarity between a shape to be detected by a trained model and a shape detected by inference.

[0033] (1: Configuration of candidate teacher data DB221) Fig. 4 is a data table showing an example of the configuration of the teacher data candidate DB221 stored in the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 4, the teacher data candidate DB221 includes information on, for example, "data ID", "data type", "data set name", "data format", "data quantity", "description", and "storage location", and stores images, videos, etc. that are candidates for teacher data for each data ID.

[0034] The column of "Data ID" of the teacher data candidate DB 221 stores identification information assigned to each group of the data DAT input to the input unit 21. In this example, "001" and "002" are stored in association with the two groups of data DAT, respectively.

[0035] The columns of "data type", "data set name", "data format", and "data quantity" of the training data candidate DB221 store basic information of the data DAT associated with each data ID.

[0036] The column "data type" of the teacher data candidate DB 221 indicates the data type of the data DAT. In this example, the data types "image" and "video" are stored in association with the data IDs "001" and "002", respectively.

[0037] The column "dataset name" in the teacher data candidate DB221 indicates the name of a group of data DAT. Any name may be stored as the "dataset name" in the teacher data candidate DB221. For example, the "dataset name" stores a name that is a sequence of the date on which the data DAT was acquired and a name associated with the data DAT. In this example, the dataset names "2023 / 07 / 01XX" and "2023 / 07 / 02YY" are stored in association with the data IDs "001" and "002", respectively.

[0038] The column "data format" of the teacher data candidate DB 221 indicates the data format of the data DAT. In this example, the data formats "PNG" and "MP4" are stored in association with the data IDs "001" and "002", respectively.

[0039] The column "data quantity" of the teacher data candidate DB 221 indicates the quantity of data DAT. In this example, data quantities "50 sheets" and "1:50:02" are stored in association with data IDs "001" and "002", respectively. In this way, the type of information stored in "data quantity" can be changed appropriately depending on the data type.

[0040] The "description" column of the training data candidate DB 221 is information added for the user's reference, and stores information for determining what kind of scene the training data is suitable for.

[0041] The column "storage location" of the teacher data candidate DB 221 indicates information on the storage location of the data DAT associated with each data ID. In this example, storage locations "AAA" and "BBB" are stored in association with data IDs "001" and "002", respectively.

[0042] (2: Configuration of teacher data DB222) Fig. 5 is a data table showing an example of the configuration of the teacher data DB 222 stored in the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 5, the teacher data DB 222 includes information on, for example, "teacher data ID", "data set name", "description", and "data quantity", and stores images as teacher data for each teacher data ID.

[0043] The column of "Teacher Data ID" in the teacher data DB 222 is identification information including a group of teacher data associated with specific tag information. In this example, "001" and "002" are stored as the "Teacher Data ID" in association with two pieces of tag information, respectively.

[0044] The columns of "data set name," "description," and "quantity of data" in the teacher data DB 222 store basic information associated with each teacher data ID.

[0045] The "dataset name" column of the teacher data DB222 indicates the name of the tag information. Any name can be stored as the "dataset name" of the teacher data DB222. In this example, the dataset names "pothole_AA" and "expressway_nega" are stored in association with the teacher data IDs "001" and "002", respectively. "pothole_AA" corresponds to one of the levels of potholes that have occurred on the road. "expressway_nega" corresponds to the case where no abnormality has occurred on the road.

[0046] The "explanation" of the teacher data DB222 is information added for user reference, and stores information indicating what kind of teacher data it is. In this example, the explanations "Pothole (AA)" and "No abnormality in the highway road surface" are stored in association with the teacher data IDs "001" and "002", respectively.

[0047] The "data quantity" in the teacher data DB 222 indicates the quantity of image data included in the associated teacher data ID. In this example, the data quantities "50 sheets" and "200 sheets" are stored in association with the data IDs "001" and "002", respectively.

[0048] 5 also shows two image data (file names "20230701090001.png" and "20230702150002.jpg") stored in the teacher data DB222 in association with the teacher data ID "001". The file name of the image data stored as teacher data in the teacher data DB222 is based on the file name of the image data to be copied from in the teacher data candidate DB221. The learning unit 24 can obtain appropriate data from the teacher data DB222 based on the specified learning settings, and create a learning teacher data file 223 by arbitrarily combining data sets to which different tag information has been added.

[0049] In this example, the teacher data DB222 stores "data set name", "description", and "quantity of data" as basic information, but is not limited to this. The teacher data DB222 may have other information as basic information, or some of this information may be omitted. The teacher data DB222 may aggregate images of the same data format, or may aggregate images of different data formats.

[0050] (3: Configuration of learning teacher data file 223) Fig. 6 is a data table showing an example of the configuration of the learning teacher data file 223 stored in the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 6, the learning teacher data file 223 includes information on, for example, "type" and "tag information", and stores images as teacher data for each type. Note that the "type" and "tag information" of the learning teacher data file 223 may be called "teacher information".

[0051] The "type" column of the learning teacher data file 223 stores information based on the settings (learning settings) when creating a learning model. In the learning settings, two or more types can be set. In this example, "posi" and "nega" are stored as "type" in association with the two types. "posi" corresponds to, for example, a case where an abnormality has occurred on the road. "nega" corresponds to, for example, a case where no abnormality has occurred on the road.

[0052] The "tag information" column of the training teacher data file 223 stores information added for user reference and based on the training settings. In this example, the tag information "Pothole_AA" and "- (no abnormality)" are stored in association with the data IDs "001" and "002", respectively.

[0053] 6 also shows two teacher image data (file names "20230701090001.jpg" and "20230702150002.jpg") associated with the type "posi" and stored in the training teacher data file 223. The file name of the image data stored as teacher data in the training teacher data file 223 is based on the file name of the image data to be copied from in the teacher data candidate DB 221. The learning unit 24 can perform weakly supervised learning based on the teacher information and teacher images stored in the training teacher data file 223.

[0054] (4: Configuration of test data DB224) Fig. 7 is a data table showing an example of the configuration of the test data DB 224 stored in the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 7, the test data DB 224 includes information on, for example, a "test data ID", a "data type", a "data set name", a "data format", a "quantity of data", and an "explanation", and stores images, videos, etc. that are targets for inference for each test data ID.

[0055] The column of "test data ID" of the test data DB224 stores, for example, identification information assigned to each group of the data DAT input to the input unit 21, similar to the data ID of the teacher data candidate DB221. In this example, "001" and "002" are stored in association with the two groups of data DAT, respectively.

[0056] The columns of "data type", "data set name", "data format", and "data quantity" of the test data DB224 store basic information of the data DAT associated with each data ID, similar to the teacher data candidate DB221. The "description" of the test data DB224 is information added for the user's reference, and stores information indicating what kind of scene the test data is.

[0057] 7 also shows two image data (file names "20230701090002.png" and "20230702150003.jpg") associated with the test data ID "001" and stored in the test data DB 224. The inference unit 26 can select test data stored in the test data DB 224 according to the purpose of the inference, and perform inference targeting the selected test data.

[0058] (5: Configuration of inference result DB225) Fig. 8 is a data table showing an example of the configuration of the inference result DB 225 stored in the data storage unit 22 included in the learning device 1 according to the first embodiment. As shown in Fig. 8, the inference result DB 225 includes information on, for example, an "inference ID", an "applied model ID", and a "test data ID", and stores the inference result for each inference ID.

[0059] The "inference ID" column of the inference result DB 225 stores identification information of the inferences performed by the inference unit 26. In this example, "001" and "002" are stored in association with the two inferences performed, respectively.

[0060] The column "applied model ID" in the inference result DB 225 stores identification information of the trained model used in the inference associated with each inference ID. In this example, the applied model IDs "Model_02" and "Model_01" are stored in association with the inference IDs "001" and "002", respectively.

[0061] The "test data ID" column of the inference result DB 225 stores the test data ID that was the subject of inference associated with each inference ID. In this example, the test data IDs "001" and "002" are stored in association with the inference IDs "001" and "002", respectively. That is, this example shows a case where inferences using different trained models are performed on images with the same test data ID.

[0062] FIG. 8 also shows two inference results stored in the inference result DB 225 in association with the inference ID "001". The table showing the inference results stores, for example, an "inference ID", an "original image", a "detection image", "detection data", and a "maximum score value". The "original image" column stores the file name of the image that was the subject of inference. The "detection image" column stores the file name of the detection image in which the inference result is superimposed on the original image. The detection image corresponds to an anomaly score map showing the degree of anomaly for each pixel. That is, the learning device 1 outputs an "anomaly score map showing the degree of anomaly for each pixel" as the inference result. The "detection data" column stores the file name of the text data (detection data) in which the detection score is recorded. The detection score is a numerical value indicating the likelihood corresponding to the detection result. The "maximum score value" column stores, for example, the numerical value of the maximum detection score among the detection results of the image.

[0063] In this example, the detection image "result_2023070109002.png", the detection data "result_2023070109002.txt", and the maximum score value "0.952" are stored in association with the original image "20230701090002.png". The detection data "result_2023070109002.txt" stores, for example, "0, 0, 0, 0.952, 0.8, 0, ...". In this example, each numerical value separated by a comma in the detection data corresponds to a detection score. The detected image "result_20230702150003.jpg", the detected data "result_20230702150003.txt", and the maximum score value "0.423" are stored in association with the original image "20230702150003.jpg".

[0064] In this way, the inference unit 26 creates the file name of the detection image by using the file name of the original image. In this example, the inference unit adds "result_" to the file name of the original image to create the file name of the detection image. Although not shown in the figure, the data of the detection image is stored in the inference result DB 225. The user can perform various tasks based on the inference results stored in the inference result DB 225. The user can also perform accuracy verification based on the results stored in the inference result DB 225.

[0065] In the above description, the learning device 1 outputs both the detection image and the detection data, but the present invention is not limited to this. The learning device 1 may be configured to output at least one of the detection image and the detection data as an inference result. The detection image may be used, for example, when the inference result is processed by visual confirmation in the later processing. The detection data may be used, for example, when the inference result is processed mechanically rather than visually in the later processing. The learning device 1 may express the position of the anomaly in black and white binary or grayscale based on the detection data. The learning device 1 may also be configured to generate a detection image based on the detection data based on a user's instruction.

[0066] <1-1-4> Configuration of model storage unit 25 9 is a block diagram showing an example of the configuration of the model storage unit 25 included in the learning device 1 according to the first embodiment. As shown in FIG. 9, the model storage unit 25 stores, for example, a trained model DB 251.

[0067] The trained model DB 251 stores trained models to be used in the inference processing function of the inference unit 26. In other words, the trained model DB 251 stores trained models generated by a weakly supervised learning function.

[0068] (Configuration of trained model DB251) Fig. 10 is a data table showing an example of the configuration of the trained model DB 251 stored in the model storage unit 25 included in the learning device 1 according to the first embodiment. As shown in Fig. 10, the trained model DB 251 includes information on, for example, a "model ID", a "creation date", a "generation parameter", a "description", and a "storage location", and stores trained models for each model ID.

[0069] The "model ID" column of the trained model DB 251 stores identification information of the trained models created by the training unit 24. In this example, "001" and "002" are stored in association with the two trained models, respectively.

[0070] The "model name" column of the trained model DB 251 stores the names of trained models created by the training unit 24. In this example, the model names "pothole detection" and "crack detection" are stored in association with the model IDs "001" and "002", respectively.

[0071] The "Creation Date" column of the trained model DB 251 stores information about the creation date when the associated trained model was created. In this example, the creation dates "2023 / 07 / 03" and "2023 / 07 / 05" are stored in association with the model IDs "001" and "002," respectively.

[0072] The “generation parameters” column of the trained model DB 251 stores information about the learning settings used when generating the learning model. In this example, the generation parameters “default” and “SET1” are stored in association with the model IDs “001” and “002”, respectively.

[0073] The “description” column of the trained model DB251 is information added for user reference and stores information explaining what purpose the trained model is used for.

[0074] The "storage location" column of the trained model DB 251 indicates information on the storage location of the trained model associated with each model ID. In this example, the storage locations "AAA" and "BBB" are stored as the "storage location" in association with the model IDs "001" and "002," respectively.

[0075] The inference unit 26 may select a trained model stored in the trained model DB 251 depending on the purpose of the inference, and perform the inference.

[0076] <1-2> Operation Next, an operation of the learning device 1 will be described as a learning method according to the first embodiment. The operation of the learning device 1 described below can be realized by the processor 11 executing a learning program stored in the memory device 12.

[0077] <1-2-1> Overview of image recognition AI using weakly supervised learning First, we will provide an overview of image recognition AI using weakly supervised learning.

[0078] In weakly supervised learning, a learning model is created based on a teaching method based on a "mostly correct" labeling technique. For example, in weakly supervised learning, tag information is added on an image-by-image basis. In weakly supervised learning, an image with normal tag information added is treated as if all pixels are normal. In weakly supervised learning, an image with abnormal (deformed) tag information added is treated as if it contains at least one abnormal pixel.

[0079] Then, in weakly supervised learning, an AI model is created that outputs an anomaly score map that indicates the degree of anomaly for each pixel based on an input image without anomalies and an input image with anomalies. For example, in an image recognition AI using weakly supervised learning, the maximum value of the anomaly score is trained to match the correct label. For example, if the maximum value is 0.01, the correct label for each image is treated as normal (0). On the other hand, if the maximum value is 0.99, the correct label for each image is treated as abnormal (1). Since the anomaly score map is generated pixel by pixel, the location of anomalies (anomalies) in the image can be visualized.

[0080] As described above, weakly supervised learning can train an image recognition AI model that can detect the location of anomalies from image-by-image teaching. In other words, an image recognition AI that uses weakly supervised learning can detect the location of anomalies simply by learning to distinguish between the presence and absence of anomalies. Because this type of image recognition AI does not require special teaching techniques, the work time required to generate training data is reduced, and the teaching work time per image can be significantly shortened.

[0081] <1-2-2> Operation of input unit 21 11 is a flowchart showing an example of the operation of the input section 21 included in the study device 1 according to the first embodiment. Hereinafter, an example of the operation of the input section 21 in the first embodiment will be described with reference to FIG.

[0082] When the input unit 21 receives data DAT from an external server or an internal storage device 13, the input unit 21 starts a series of processes in FIG. 11 (START).

[0083] First, the input unit 21 collects data DAT that are candidates for teacher data (S10). The data DAT collected in the process of S10 is a video, an image, a video captured by a camera, etc. The data format of the data DAT that can be collected by the input unit 21 is not particularly limited. The data DAT may be handled by the learning device 1 and may include images suitable for learning.

[0084] Then, the input unit 21 stores the collected data DAT in the teacher data candidate DB 221 (S11). Note that the input unit 21 may store a part of the collected data DAT in the test data DB 224. The input unit 21 may store the collected data DAT in at least one of the teacher data candidate DB 221 and the test data DB based on the data definition.

[0085] When the process of S11 is completed, the input unit 21 ends the series of processes in FIG.

[0086] <1-2-3> Operation of the data processing unit 23 (1: 1st action) Fig. 12 is a flowchart showing an example of a first operation of the data processing unit 23 included in the learning device 1 according to the first embodiment. The first operation corresponds to preparation processing before the start of annotation, and corresponds to processing by the data confirmation function of the data processing unit 23. Below, an example of the first operation of the data processing unit 23 in the first embodiment will be described with reference to Fig. 12.

[0087] When the data processing unit 23 receives an instruction to start annotation using a data set selected based on a user operation, the data processing unit 23 starts a series of processes in FIG. 12 (start).

[0088] First, the data processing unit 23 acquires a data set (S20) from the teacher data candidate DB 221. In the process of S20, data corresponding to the purpose of creating the teacher data is extracted from the videos and images stored in the teacher data candidate DB 221.

[0089] Next, the data processing unit 23 checks the data format of the acquired data set (S21). If the data format of the acquired data set is a moving image (S21: moving image), the data processing unit 23 proceeds to the process of S22. If the data format of the acquired data set is an image (S21: image), the data processing unit 23 proceeds to the process of S23.

[0090] In the process of S22, the data processing unit 23 executes an image extraction process. The image extraction process is a process of extracting images under predetermined conditions from a moving image corresponding to the acquired data set. For example, in the image extraction process, frame images included in the moving image are extracted at predetermined intervals. In other words, in the image extraction process, some of the frames included in the moving image are thinned out, and the remaining frame images are stored. When the process of S22 ends, the data processing unit 23 proceeds to the process of S23.

[0091] In the process of S23, the data processing unit 23 excludes images that are not suitable for teacher data from the images related to the acquired data set. In other words, in the process of S23, the data processing unit 23 saves images that can be used for teacher data, that is, images that are suitable as teacher data, from the images related to the acquired data set. In the process of S23, the data processing unit 23 regards, for example, an image in which a specific object is detected as an image that is not suitable for teacher data. For example, a stop sign is an example of such a specific object.

[0092] When the process of S23 ends, the data processing unit 23 executes the annotation tool (S24). The function of the annotation tool is realized by the teacher data creation function of the data processing unit 23. By executing the annotation tool, the user can tag the image selected by the process of S23. An example of a specific process of the annotation tool will be described with reference to FIG. 13.

[0093] When the process of S24 is completed, the data processing unit 23 ends the series of processes in FIG.

[0094] In the above description, the data processing unit 23 removes images that are not suitable for teacher data in the process of S23, but the present invention is not limited to this. The process of S23 may be omitted. In addition, the selection of images suitable for teacher data in the process of S23 may be performed by a user. The user may select images suitable for teacher data using a data confirmation function of the data processing unit 23. The data confirmation function of the data processing unit 23 may have functions such as playback, rewinding, and fast forwarding to easily search for frame images that are the subject of teacher data when processing a video, or may have a function to display frame images by specifying a frame number. The data confirmation function of the data processing unit 23 may have a function to save the identified frame image and call up a teacher data creation function.

[0095] (2: 2nd action) 13 is a flowchart showing an example of a second operation of the data processing unit 23 included in the learning device 1 according to the first embodiment. The second operation corresponds to processing of the teacher data creation function of the data processing unit 23 when the annotation tool is executed. Hereinafter, an example of the second operation of the data processing unit 23 in the first embodiment will be described with reference to FIG.

[0096] The data processing unit 23 starts the series of processes in FIG. 13 based on the execution of the annotation tool (START).

[0097] First, the data processing unit 23 loads an image (S30). In the process of S30, one image is selected from the images selected in the first operation. The loaded image is displayed on the display device 14, for example, and becomes available for the user to refer to.

[0098] Next, data processing unit 23 accepts the selection of a tag (S31). The user refers to the image displayed on display device 14 and specifies tag information suitable for the target image. In other words, the user instructs the selected image by specifying tag information.

[0099] Next, the data processing unit 23 detects a teacher data save operation (S32). The teacher data save operation corresponds to, for example, clicking a predetermined button on the annotation tool or inputting a predetermined shortcut key on the keyboard.

[0100] Next, the data processing unit 23 checks whether or not a folder associated with the selected tag exists (S33). In the process of S33, the data processing unit 23, for example, refers to a folder in the teacher data DB 222 to check whether or not a folder with the same name as the selected tag exists. If a folder associated with the selected tag does not exist (S33: NO), the data processing unit 23 proceeds to the process of S34. If a folder associated with the selected tag exists (S33: YES), the data processing unit 23 proceeds to the process of S35.

[0101] In the process of S34, the data processing unit 23 creates a new folder associated with the selected tag. The name of the folder newly created by the data processing unit 23 is based on the name of the selected tag. For example, the data processing unit 23 names the newly created folder the same as the selected tag. When the process of S34 ends, the data processing unit 23 proceeds to the process of S35.

[0102] In the process of S35, the data processing unit 23 saves the image in the folder associated with the selected tag. This causes the tag selected in the process of S31 to be associated with the current image. The current image data is copied and saved in the folder associated with the selected tag.

[0103] Next, the data processing unit 23 checks whether annotation of all images in the data set is complete (S36). If annotation of all images in the data set is not complete, the data processing unit 23 loads the next image (S37) and proceeds to the process of S31. If annotation of all images in the data set is complete, the data processing unit 23 ends the series of processes in FIG. 13 (END).

[0104] By executing the second operation as described above, a folder is created in the teacher data DB 222 for each piece of tag information. Then, the image stored in the folder associated with the tag information is saved as teacher data. The process of S36 may be replaced with other processes. For example, the process of S36 may be replaced with a selection operation of the next image or an end operation of the annotation tool by the user. In this case, based on the selection operation of the next image being performed by the user, the data processing unit 23 loads the next image and proceeds to the process of S31. Based on the end operation of the annotation tool being performed by the user, the data processing unit 23 ends the series of processes in FIG. 13.

[0105] (3: A concrete example of the allocation process of training data) Fig. 14 is a schematic diagram showing a specific example of a teacher data allocation process by the data processing unit 23 included in the learning device 1 according to the first embodiment. The teacher data allocation process by the data processing unit 23 corresponds to, for example, the processes of S31 to S35 shown in Fig. 13. As shown in Fig. 14, a predetermined data set of the teacher data candidate DB221 includes, for example, image data IMG1 to IMG4. The teacher data DB222 includes, for example, folders F1 and F2 associated with tags T1 and T2, respectively, and a folder F3 associated with undetected.

[0106] The user operates the annotation tool to tag each of the image data IMG1 to IMG4. For example, if tag T1 is selected when referring to image data IMG1, the data processing unit 23 copies the image data IMG1 to folder F1 by processing in S35. If tag T2 is selected when referring to image data IMG2, the data processing unit 23 copies the image data IMG2 to folder F2 by processing in S35. If no abnormality is detected when referring to image data IMG3, the data processing unit 23 copies the image data IMG3 to folder F3 by processing in S35. If no abnormality is detected when referring to image data IMG4, the data processing unit 23 copies the image data IMG4 to folder F3 by processing in S35. If the destination folder does not exist, the data processing unit 23 creates a new folder as appropriate.

[0107] (4: A concrete example of the annotation tool GUI30) 15 is a schematic diagram showing a specific example of GUI 30 of the annotation tool displayed on display device 14 included in study device 1 according to the first embodiment. As shown in FIG. 15, GUI 30 of the annotation tool in the first embodiment includes multiple areas 31-37.

[0108] Area 31 displays the address of a folder in the teacher data candidate DB 221 in which the data set input to the annotation tool is saved, i.e., the input folder. Area 32 displays the address of a folder in the teacher data DB 222 to which the tagging results using the annotation tool are output, i.e., the output folder. Area 33 corresponds to, for example, a button used to select an output folder. Area 34 corresponds to a button used to switch the image displayed in area 37. The functions assigned to the buttons in area 34 can be changed as appropriate depending on the design of the GUI 30 of the annotation tool.

[0109] The area 35 displays a list of tag information. Tag information included in the list of tag information may be added or deleted by a user operation. The list of tag information may include tag information corresponding to non-detection. In this example, "None" corresponds to tag information corresponding to non-detection of anomaly. "None" may be selected when loading an image. When tag information corresponding to non-detection of anomaly is not displayed in the area 35, the unselected state corresponds to non-detection of anomaly.

[0110] Area 36 displays a list of image data included in the data set acquired when the annotation tool was launched. A user can change the image data displayed in area 37 by selecting any image data displayed in the list of image data. Area 37 displays image data that is currently being tagged. Area 38 corresponds to a button used to save a copy of the current image in a folder associated with selected tag information or unselected information.

[0111] For example, when the user finds an abnormal area DA in the image data displayed in area 37, the user selects a tag corresponding to the found abnormal area DA from the list of tag information. After selecting the tag, the user clicks a button in area 38. This allows the user to copy the image data to a folder associated with the tag information. Note that if the button in area 38 is clicked with no tag information selected, the image data is copied to a folder associated with undetected tag information.

[0112] As described above, the teacher data creation function of the data processing unit 23 has the ability to display the image that is the subject of the teacher data, select the tag information to be taught, and save the teacher data, all with a simple operation such as a single click of the mouse.

[0113] <1-2-4> Operation of the learning unit 24 16 is a flowchart showing an example of the operation of the learning unit 24 included in the learning device 1 according to the first embodiment. Hereinafter, an example of the operation of the learning unit 24 in the first embodiment will be described with reference to FIG.

[0114] When the learning unit 24 receives an instruction to create a learning model using learning settings designated based on a user operation, the learning unit 24 starts a series of processes in FIG. 16 (START).

[0115] First, the learning unit 24 acquires the settings of a learning model to be created (S40). That is, in the process of S40, the learning unit 24 reads out detailed parameters of the learning settings designated by the user.

[0116] Next, the learning unit 24 creates folders for each judgment type based on the acquired settings (S41). In the process of S41, the learning unit 24 creates folders corresponding to at least two types of types: no shape (nega) and shape (posi).

[0117] Next, the learning unit 24 acquires a teacher image from the teacher data DB 222 based on the acquired settings (S42). That is, in the process of S42, the learning unit 24 extracts a teacher image from the teacher data DB 222 according to the characteristics of the learning model to be created.

[0118] Next, the learning unit 24 distributes the teacher images to folders based on the acquired settings (S43). Specifically, in the process of S42, the learning unit 24 copies image data to which tag information associated with the learning settings is added to the multiple folders created in the process of S41. That is, the extracted teacher images are distributed to folders associated with the same tag information as the teacher images themselves according to the characteristics of the learning model to be created.

[0119] Next, the learning unit 24 uses the folder to which the teacher images are sorted to create a learning teacher data file 223 (S44). That is, by the process of S44, the teacher images for each type based on the learning settings are converted into a data format suitable for weakly supervised learning.

[0120] Next, the learning unit 24 executes a learning process using the created learning teacher data file 223 (S45). The process of S45 corresponds to a weakly supervised learning process in which parameters are specified based on the learning settings and teacher data taught on a pixel-by-pixel basis is not required.

[0121] Next, the learning unit 24 stores the generated trained model in the trained model DB 251 (S46).

[0122] When the process of S46 is completed, the learning unit 24 ends the series of processes in FIG. 16 (END).

[0123] As described above, the learning preparation function of the learning unit 24 creates a folder for each type of shape to be detected or for the presence or absence of a shape to be detected, without specifying the position of the shape to be detected. Then, the learning preparation function of the learning unit 24 converts the acquired teacher image into a learning teacher data file 223 using a folder for each type of shape, without adding special information (e.g., tag information, etc.). At this time, the learning unit 24 generates the learning teacher data file 223 using at least two folders out of the multiple folders associated with tag information in the teacher data DB 222. Thereby, the weakly supervised learning function of the learning unit 24 can generate a learning model based on the created learning teacher data file 223.

[0124] (Specific example of teacher data sorting process) FIG. 17 is a schematic diagram showing an example of a process of allocating training teacher data by the learning unit 24 included in the learning device 1 according to the first embodiment. The process of allocating training teacher data by the inference unit 26 corresponds to, for example, the processes of S41 to S43 shown in FIG. 16. As shown in FIG. 17, the training teacher data DB 222 includes, for example, image data IMG1 to IMG4. The training teacher data file 223 is associated with, for example, a folder F4 associated with "detection" and a folder F5 associated with "undetected". In this example, a case is illustrated in which a user creates the training teacher data file 223 using a learning setting in which tags T1 and T2 are assigned to "detection" and undetected abnormality is assigned to "undetected".

[0125] In this example, the image data IMG1 to IMG4 are associated with tag information similar to the example shown in FIG. 14. In this case, in the process of S43, the inference unit 26 copies the image data IMG1 and IMG2 to the folder F4, and copies the image data IMG3 and IMG4 to the folder F5. Then, the inference unit 26 generates the learning teacher data file 223 using the folders F4 and F5 in which the image data based on the learning settings are saved. Note that, if the copy destination folder does not exist, the inference unit 26 creates a new folder as appropriate. As the learning settings, multiple pieces of tag information may be classified into three or more items. Image data associated with tag information may be copied to undetected.

[0126] <1-2-5> Operation of the inference unit 26 18 is a flowchart showing an example of the operation of the inference unit 26 included in the learning device 1 according to the first embodiment. Hereinafter, with reference to FIG. 18, an example of the operation of the inference unit 26 in the first embodiment will be described.

[0127] When the inference unit 26 receives an instruction to start inference using the test data and the trained model selected based on a user operation, the inference unit 26 starts the series of processes in FIG. 18 (start).

[0128] First, the inference unit 26 acquires (S50) a data set stored in the test data DB 224. The data set acquired in the process of S50 corresponds to a data set selected by the user.

[0129] Next, the inference unit 26 selects a trained model to be used for inference from the trained model DB 251 (S51). The trained model selected in the process of S51 corresponds to the trained model selected by the user.

[0130] Next, the inference unit 26 executes an inference process on the acquired data set (S52). In the process of S52, the inference unit 26 executes an inference process based on the data type (e.g., video or image) and parameters specified by the user and the trained model selected in the process of S51.

[0131] Next, the inference unit 26 stores the inference result in the inference result DB 225 (S53). That is, in the process of S53, the inference result DB 225 stores the inference result image and the inference score obtained in the process of S52.

[0132] When the process of S53 is completed, the inference unit 26 ends the series of processes in FIG. 18 (END).

[0133] (1: Specific examples of inference results) FIG. 19 is a schematic diagram showing a specific example of an inference result generated by the inference unit 26 included in the learning device 1 according to the first embodiment. FIG. 19 shows an example of an inference result image (detection image) in which an inference result is superimposed on an original image. As shown in FIG. 19, the inference result image shows the position of the pothole PH as a detection result of the deformation. As a method for the inference unit 26 to show the detection result of the deformation, the inference unit 26 may superimpose a line surrounding the position of the pothole PH on the original image, or may apply image processing to the portion of the pothole PH. The inference unit 26 may superimpose and display a plurality of detection results of deformations on one inference result image. In this case, the inference unit 26 may show that these are different deformations by applying different image processing to each type of deformation.

[0134] (2: Modification of the operation of the inference unit 26) Fig. 20 is a flowchart showing a modified example of the operation of the inference unit 26 included in the learning device 1 according to the first embodiment. Below, a modified example of the operation of the inference unit 26 in the first embodiment will be described with reference to Fig. 20. In this modified example, it is assumed that a camera is externally connected to the learning device 1 or is built in.

[0135] When the inference unit 26 detects the start of an application for starting inference using the camera image, it starts the series of processes in FIG. 20 (start).

[0136] First, the inference unit 26 executes a camera connection process (S60). Note that if the camera is already connected, the process of S60 may be omitted.

[0137] Next, the inference unit 26 selects a trained model to be used for inference from the trained model DB 251 (S61). The trained model selected in the process of S61 corresponds to the trained model selected by the user.

[0138] Next, the inference unit 26 detects an instruction to start inference (S62). The instruction to start inference is issued by a user operation. The user can generate an instruction to start inference by operating the input interface 15 at the timing when the user wants to start inference using the camera image.

[0139] Next, the inference unit 26 performs inference processing on the image captured by the camera (S63). In the processing of S63, the inference unit 26 may perform inference processing on all frame images of the camera's video, or may perform the image extraction processing (S22) described with reference to Fig. 12 and perform inference processing on the extracted images.

[0140] Next, similarly to the process of S53, the inference unit 26 stores the inference result in the inference result DB 225 (S64). The process of S64 may be executed every time an inference result is generated, or may be executed based on the generation of a predetermined amount of inference results.

[0141] Next, the inference unit 26 checks whether or not an inference stop instruction has been detected (S65). The inference stop instruction is issued by a user operation. If an inference stop instruction has not been detected (S65: NO), the inference unit 26 proceeds to the process of S63. That is, the inference unit 26 executes inference processing for the next image captured by the camera. If an inference stop instruction has been detected (S65: YES), the inference unit 26 ends the series of processes in FIG. 20 (END).

[0142] <1-3> Effects of the first embodiment According to the learning device 1 according to the first embodiment, it is possible to efficiently create teacher data for a weakly supervised learning model. The effects of the first embodiment will be described in detail below.

[0143] Image recognition AI can be used to improve the efficiency of detecting abnormalities during road inspections. In road inspections, road managers may have a large amount of candidate training data, such as images from past inspection records and videos taken from inspection vehicles. The amount of data for these candidate training data is expected to increase in the future. For example, in road inspections, it is important to detect the location of abnormalities in images in order to carry out prompt repairs. A known method for detecting the location of abnormalities in images is to prepare training data that teaches the location of abnormalities in pixel units for a large number of learning images.

[0144] In an annotation tool that performs pixel-by-pixel teaching, for example, the following processes (1) to (5) are executed: (1) select one image, (2) confirm the anomaly to be taught, (3) specify the anomaly by enclosing it with a mouse click, (4) specify a tag (label) (by text input or by selecting from a list), and (5) save the tag information. In this case, the tag information is saved in a file separate from the image. In this case, the image and annotation file need to be managed as a set. In other words, one tag information file is prepared for each image. Since the tag information file is saved in a file separate from the image, the image and annotation file need to be input as a set during learning. Then, by learning a large number of normal images, abnormal images, and pixel-by-pixel labels, an image recognition AI that can detect the presence or absence of pixel-by-pixel anomalies can be realized. However, since pixel-by-pixel teaching takes time and cost, it is preferable to create a trained model by image-by-image teaching, as in weakly supervised learning.

[0145] In contrast, the annotation tool of the first embodiment is capable of creating teacher data for weakly supervised learning, and executes the following processes (1) to (4): (1) select one image, (2) confirm the anomaly to be taught, (3) specify a tag (label), and (4) save as teacher data. In this case, teacher data is generated without pixel-by-pixel teaching. Furthermore, a tag information file is not required, and images that can be used as teacher data are stored in folders for the specified tag units. During learning, folders such as "deformation A," "deformation B," and "no deformation" are created, and the created teacher data is stored in them.

[0146] Thus, in the first embodiment, there is no tag information file associated with the teacher data, and the image file itself does not have tag information. The folder in the teacher data DB222 itself plays the role of tag information. Also, for example, an operation such as using image A as "deformation A" in the first learning and as "no deformation" in the second learning can be realized by simply moving the image file. Therefore, the annotation tool of the learning device 1 according to the first embodiment can simplify the creation of various models and learning trials, and can efficiently create teacher data in a weakly supervised learning model.

[0147] In weakly supervised learning, even if a road surface abnormality detection AI trained on a public dataset for general roads is applied to highway images, there is a risk of missed detections and a large number of false alarms. In other words, it is difficult to apply a trained model based on weakly supervised learning to a site or abnormality different from the one learned. For this reason, weakly supervised learning requires re-learning with the applied site data. In response to this, by reducing the learning cost using the learning device 1 according to the first embodiment, a trained model to be used for practical inspection inference at a lower cost can be constructed. In this way, the learning device 1 according to the first embodiment can promote the introduction and business application of image recognition AI to which weakly supervised learning is applied, and can contribute to the sophistication and efficiency of business related to image recognition in various fields.

[0148] Furthermore, in the learning device 1 according to the first embodiment, since teaching information is not given for each image, the combination of teacher data for the model to be generated can be easily changed. Therefore, the learning device 1 according to the first embodiment can easily retry learning, and can build a trained model to be used for practical inference at lower cost.

[0149] <2> Second embodiment The learning device 1 according to the second embodiment has a configuration in which a function for extracting an area including a deformation by a simple operation is added to the annotation tool described in the first embodiment. Details of the learning device 1 according to the second embodiment will be described below, focusing on the differences from the first embodiment.

[0150] <2-1> Configuration The hardware configuration of the learning device 1 according to the second embodiment is the same as that of the first embodiment. The functional configuration of the learning device 1 according to the second embodiment is the same as that of the first embodiment, except that the function of the data processing unit 23 is added.

[0151] The teacher data creation function of the data processing unit 23 of the learning device 1 according to the second embodiment has a function that allows display of the image to be the subject of the teacher data, selection of tag information to be taught, selection of the image cut-out range, and storage of the teacher data to be completed with a simple operation such as one click of the mouse. Selection of the image cut-out range is performed only when necessary. By setting the cut-out size in advance, the user can specify the cut-out range with one click of the mouse. Furthermore, when creating teacher data from an inference result image, the teacher data creation function in the second embodiment may have a function of obtaining the inference result image and the original image from the data storage unit 22 and displaying both images simultaneously, making it easy to identify the image cut-out range.

[0152] <2-2> Operation The operation of the learning device 1 according to the second embodiment is similar to that of the first embodiment, except for the operation related to the data processing unit 23.

[0153] 21 is a flowchart showing an example of the second operation of the data processing unit 23 included in the learning device 1 according to the second embodiment. Hereinafter, with reference to FIG. 21, an example of the second operation of the data processing unit 23 in the second embodiment will be described.

[0154] In the second operation of the data processing unit 23 in the second embodiment, some of the processes executed between S30 and S36 are different from the second operation of the data processing unit 23 described in the first embodiment using FIG. 13.

[0155] Specifically, after the process of S30, the data processing unit 23 accepts the selection of a tag (S31), similarly to the first embodiment.

[0156] Next, the data processing unit 23 accepts the selection of the teaching position (S70). In the processing of S70, the data processing unit 23 can detect the image cut-out range (teaching position) indicated by the user's operation. The processing of S70 allows the user to perform teaching on the selected image. If the image cut-out range has been set in advance, the user can specify the cut-out range with one click on the screen.

[0157] Next, the data processing unit 23 detects a teacher data save operation (S32), similarly to the first embodiment.

[0158] Next, the data processing unit 23 checks whether or not a folder associated with the selected tag exists (S33), as in the first embodiment. If a folder associated with the selected tag does not exist (S33: NO), the data processing unit 23 proceeds to processing of S34. If a folder associated with the selected tag exists (S33: YES), the data processing unit 23 proceeds to processing of S71.

[0159] In the process of S34, the data processing unit 23 creates a new folder associated with the selected tag, similarly to the first embodiment. After completing the process of S34, the data processing unit 23 proceeds to the process of S71.

[0160] In the process of S71, the data processing unit 23 checks whether or not a teaching position has been selected. The data processing unit 23 checks whether or not a teaching position has been selected based on whether or not the user has performed a teaching position selection operation in the process of S70. If it is not confirmed that a teaching position has been selected (S71: NO), the data processing unit 23 proceeds to the process of S35. If it is confirmed that a teaching position has been selected (S71: YES), the data processing unit 23 proceeds to the process of S72.

[0161] In the process of S35, the data processing unit 23 saves the image in a folder associated with the selected tag, similarly to the first embodiment. When the process of S35 is completed, the data processing unit 23 proceeds to the process of S36.

[0162] In the process of S72, the data processing unit 23 renames and saves the cropped image based on the teaching position in the folder associated with the selected tag (S72). For example, a file name of the cropped image is a file name of the original image to which information indicating the teaching position is added. When the process of S72 is completed, the data processing unit 23 proceeds to the process of S36.

[0163] (1: Specific example of teacher data DB222) Fig. 22 is a data table showing an example of the configuration of the teacher data DB222 stored in the data storage unit 22 included in the learning device 1 according to the second embodiment. As shown in Fig. 22, the teacher data DB222 in the second embodiment is obtained by adding information about the file name of the cropped image, the original file name, and the teaching position to the teacher data DB222 described in the first embodiment using Fig. 5.

[0164] In this example, when the teaching position for the original image (20230701090001.png) is (100,120), the file name of the cropped image is set to "20230701090001_100_120.png". When the teaching position for the original image (20230702150020.jpg) is (600,800), the file name of the cropped image is set to "20230702150020_600_800.png". That is, in this example, "_(X coordinate)_(Y coordinate)" is added before the extension of the file name of the original image. Note that the "X coordinate" corresponds to the position of the pixels arranged in the X direction in the image, and the "Y coordinate" corresponds to the position of the pixels arranged in the Y direction in the image.

[0165] In this way, in the second embodiment, the image used as the teacher data has the name of the original image and the cut-out range from the original image embedded in the image name. This allows the relationship with the original image to be reproduced in the cropped image based on the teaching position. The file name of the cropped image may be any name as long as it can indicate the teaching position. The file may be renamed according to rules different from those described above.

[0166] (2: A concrete example of the annotation tool GUI30) Fig. 23 is a schematic diagram showing an example of the GUI 30 of the annotation tool displayed on the display device 14 included in the study device 1 according to the second embodiment. As shown in Fig. 23, the GUI 30 of the annotation tool in the second embodiment has a configuration in which a plurality of areas 40 to 42 are added to the GUI 30 described in the first embodiment using Fig. 15.

[0167] Area 40 corresponds to a button for crop setting when the teaching position is indicated. When area 40 is clicked, a screen is displayed on which the crop setting can be changed by the user's operation. The crop setting includes the setting of the range for cropping the image based on the teaching position.

[0168] Area 41 indicates the range of the cropped image based on the teaching position, and is displayed, for example, in a frame visible to the user. Area 41 is generated based on selecting tag information in area 35 and then clicking on the position of the abnormality displayed in area 37. The size of area 41 can be changed based on the crop setting. Area 41 includes abnormal area DA. For example, the center part of abnormal area DA corresponds to the teaching position based on the user's operation.

[0169] Area 42 is an area that displays information in which tag information and teaching positions are combined. When a combination of tag information selection in area 35 and teaching position selection in area 37 is executed multiple times, multiple pieces of information in which tag information and teaching positions are combined are displayed in area 42. When a button in area 38 is clicked, a cropped image based on the information displayed in area 42 is saved in a folder of the associated tag information. Information added to area 42 can be deleted based on a user operation.

[0170] For example, when the user finds an abnormal area DA in the image data displayed in area 37, the user selects a tag corresponding to the found abnormal area DA from the list of tag information. The user then clicks on a portion of the image displayed in area 37 that corresponds to the abnormality. Then, information combining the coordinates of the clicked portion with the tag information is recorded and displayed in area 42. After that, the user finishes the instruction and clicks on a button in area 38. This allows the user to copy at least one cropped image to a folder associated with the tag information.

[0171] As described above, the data processing unit 23 in the second embodiment has a function that allows displaying the image to be used as the training data, selecting tag information to be taught, selecting the image cutout range, and saving the training data to be completed with a simple operation such as one click of a mouse. In the second embodiment, when training data is created from an inference result image, both the inference result image and the original image may be displayed in the area 37. In this case, the area 34 functions as a button that switches the state of the area 37 between "displaying both the inference result image and the original image side by side" and "displaying only the original image large."

[0172] Other operations of the learning device 1 according to the second embodiment are similar to those of the first embodiment.

[0173] <2-3> Effects of the second embodiment In the learning device 1 of the second embodiment, the data processing unit 23 can generate a cropped image including information of one teaching position from image data based on image data or video data obtained from the teacher data candidate DB221, and save it in a folder in the teacher data DB222 associated with tag information selected by the user.

[0174] In this way, the annotation tool of the learning device 1 according to the second embodiment can create teacher data indicating the position of the teaching target by a simple operation as necessary. As a result, the learning device 1 according to the second embodiment can create teacher data with higher learning efficiency than the first embodiment. Therefore, the learning device 1 according to the second embodiment can improve the accuracy of the trained model more than the first embodiment, in addition to the same effects as the first embodiment.

[0175] Furthermore, in the learning device 1 according to the second embodiment, original image information and position information are embedded in the file name of the training data of the cropped image. This allows the user to easily identify the original image and the specified position from the created training data. As a result, it becomes easier to evaluate the generated trained model, which contributes to quality assurance of the image recognition AI.

[0176] In the second embodiment, an operation for indicating a teaching position is added, but a cropped image to be used as teacher data can be created with a few clicks. Therefore, the annotation tool of the learning device 1 according to the second embodiment can reduce the time and cost of teaching compared to teaching in pixel units, as in the first embodiment.

[0177] <3> others In the above embodiment, the case where the image recognition AI is used for road inspection is exemplified, but is not limited thereto. The learning device 1 according to the above embodiment can also be used for infrastructure inspection such as structures and piping, accuracy improvement learning of anomaly detection AI in equipment inspection, and accuracy improvement learning of AI that detects specific patterns as part of a quality check of a product. In such cases, the same effect as in the above embodiment can be obtained. In the above embodiment, the case where the mouse is mainly used to operate the annotation tool is described, but is not limited thereto. Some operations may be realized by keyboard shortcut keys, etc. For example, tag selection and teacher data saving operations can also be performed by keyboard operations. In the second embodiment, the teaching position may be selected by the user tapping the screen when the display device 14 is a touch panel.

[0178] In the above embodiment, the flowchart used to explain the operation is merely an example. The operation explained using the flowchart may have the order of processing changed, other processing may be added, some processing may be omitted, or some processing may be executed in parallel, to the extent possible. Each of the processing described in the above embodiment may be realized by dedicated hardware. The processing described in the above embodiment may be a mixture of processing executed by software and processing executed by hardware, or may be only one of them.

[0179] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0180] 1. Learning device 11…Processor 12...Memory device 13...Storage device 14...Display device 15...Input interface 16…Communication module 21...Input section 22…Data storage section 23...Data processing unit 24…Learning section 2.0210 25…Model Storage Section 26... Reasoning part 30…GUI 31~38,40~42...area 221…Teacher data candidate DB 222…Teacher data DB 223...Teacher data file for learning 224…Test data DB 225…Inference result DB 251…Trained model DB

Claims

1. An input unit that stores image data or video data collected from an external source in a teacher data candidate database; A data processing unit that generates teacher data from image data or video data acquired from the teacher data candidate database and stores the generated teacher data in a teacher data database; a learning unit that generates a training teacher data file from the data acquired from the training teacher data database, generates a trained model by performing weakly supervised learning based on the generated training teacher data file, and stores the generated trained model in a trained model database. Learning device.

2. The data processing unit generates teacher data without pixel-by-pixel teaching. The learning device according to claim 1 .

3. The data processing unit generates teacher data by associating tag information on an image-by-image basis. The learning device according to claim 2 .

4. The data processing unit creates a plurality of folders associated with tag information in the teacher data database, and stores image data based on the image data or video data acquired from the teacher data candidate database in the folders associated with the tag information selected from the teacher data database, the learning unit generates the training data file by using at least two folders among a plurality of folders associated with the tag information; The learning device according to claim 3.

5. the learning unit creates a plurality of learning folders associated with a type of detection target, copies teacher data in a folder of associated tag information among the plurality of folders in the teacher data database for each of the types to a corresponding learning folder, and generates the learning teacher data file using the plurality of learning folders. The learning device according to claim 4.

6. The data processing unit generates a cropped image including information of one teaching position from image data based on the image data or video data acquired from the teacher data candidate database, and stores the cropped image in a folder associated with the selected tag information. The learning device according to claim 4.

7. The data processing unit embeds information for identifying the image from which the cropped image was created and information based on the teaching position in a file name of the cropped image. The learning device according to claim 6.

8. Further comprising an inference unit that performs inference on image data or video data collected from an external source using the trained model acquired from the trained model database, and stores the inference result in an inference result database. The learning device according to claim 1 .

9. Collecting image data or video data from an external source and storing it in a teacher data candidate database; Creating teacher data from the image data or the video data acquired from the teacher data candidate database, and storing the created teacher data in a teacher data database; generating a training teacher data file from the data acquired from the training teacher data database, generating a trained model by performing weakly supervised learning based on the training teacher data file, and storing the generated trained model in a trained model database. How to learn.

10. On the computer, Collecting image data or video data from an external source and storing it in a teacher data candidate database; Creating teacher data from the image data or the video data acquired from the teacher data candidate database, and storing the created teacher data in a teacher data database; generating a training teacher data file from the data acquired from the training teacher data database, generating a trained model by performing weakly supervised learning based on the training teacher data file, and storing the generated trained model in a trained model database; Study program.

Images