Learning device, learning method, and learning program

Abstract

The present invention reduces the workload of an operator when generating learning data. The learning device includes: a learning unit that generates a learning completion model by performing learning processing using learning data including each image data included in a default image data group and each true value data when each image data included in the default image data group has been subjected to segmentation processing; a collection unit that collects an x-th output image data group that is output by inputting the x-th image data group among the plurality of image data groups into the (x-1)-th learning completion model; and a generation unit that generates x-th-order learning data by acquiring a processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into each true value data, and adding the processed x-th output image data group to (x-1)-th-order learning data, the learning unit performs learning processing using the xth-order learning data to generate an xth-order learning completion model.

Description

Technical Field

[0001] The present disclosure relates to a learning device, a learning method, and a learning program. Background Art

[0002] As "object detection processing" for detecting an object from image data, conventionally, object detection processing for various tasks has been proposed. Among them, "instance segmentation processing" performs a task of assigning class labels to an object to be detected in image data at the pixel level.

[0003] Here, in order to achieve high processing accuracy in this instance segmentation processing, it is necessary to prepare a plurality of learning data in which class labels at the pixel level are correctly assigned to an object to be detected.

[0004] <Prior Art Documents>

[0005] <Patent Documents>

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2019-101535 Summary of the Invention

[0007] <Problems to be Solved by the Invention>

[0008] On the other hand, when manually generating this learning data, the work burden on the operator is high.

[0009] An object of the present disclosure is to reduce the work burden on an operator when generating learning data.

[0010] <Means for Solving the Problems>

[0011] The learning device according to the first aspect of the present disclosure includes: a learning unit that performs a learning process using learning data to generate a learned complete model, where the learning data includes each image data included in a default image data group and each ground truth data in a case where each image data included in the default image data group is subjected to a segmentation process; a collection unit that collects a first x-th output image data group output by inputting the x-th image data group in a plurality of image data groups into the (x - 1)-th learned complete model, where 1 ≤ x ≤ N and N is an integer of 2 or more; and a generation unit that generates x-th order learning data by obtaining a processed first x-th output image data group obtained by processing each output image data included in the collected first x-th output image data group into each ground truth data and adding it to the (x - 1)-th order learning data, where the learning unit uses the x-th order learning data to perform a learning process and generate an x-th order learned complete model.

[0012] The second aspect of the present disclosure is the learning device described in the first aspect, further comprising: a storage unit that stores the Nth-order learning data generated by the generation unit.

[0013] The third aspect of the present disclosure is the learning device described in the first or second aspect, wherein the number of image data included in the xth image data group is larger than the number of image data included in the default image data group.

[0014] The fourth aspect of the present disclosure is the learning device described in any one of the first to third aspects, wherein the number of image data included in the (x + 1)th image data group is larger than the number of image data included in the xth image data group.

[0015] The fifth aspect of the present disclosure is the learning device described in any one of the first to fourth aspects, wherein the ground truth data includes image data with a class label of normal fine particles assigned to pixels and image data with a class label of aggregated fine particles assigned to pixels.

[0016] The sixth aspect of the present disclosure is a learning method, which is executed by a computer of a learning device to perform the following processes: a process of generating a learned complete model by performing a learning process using learning data, where the learning data includes each image data included in the default image data group and each ground truth data in the case where each image data included in the default image data group is subjected to a segmentation process; a process of collecting the xth output image data group output by inputting the xth image data group in a plurality of image data groups into the (x - 1)th learned complete model, where 1 ≤ x ≤ N and N is an integer of 2 or more; a process of generating the xth-order learning data by obtaining the processed xth output image data group obtained by processing each output image data included in the collected xth output image data group into each ground truth data and adding it to the (x - 1)th-order learning data; and a process of generating the xth-order learned complete model by performing a learning process using the xth-order learning data.

[0017] The seventh aspect of the present disclosure is a learning program that causes a computer of a learning device to execute the following processes: a process of generating a learned complete model by performing a learning process using learning data, where the learning data includes each image data included in a default image data group and each ground truth data in a case where each image data included in the default image data group is subjected to a segmentation process; a process of collecting an x-th output image data group output by inputting an x-th image data group among a plurality of image data groups into an (x - 1)-th learned complete model, where 1 ≤ x ≤ N and N is an integer of 2 or more; a process of generating an x-th order learning data by obtaining a processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into each ground truth data and adding it to the (x - 1)-th order learning data; and a process of generating an x-th order learned complete model by performing a learning process using the x-th order learning data.

[0018] <Effects of the Invention>

[0019] According to the present disclosure, it is possible to reduce the workload of an operator when generating learning data. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Figure 1 is a diagram showing an example of the system configuration of a learning system including a learning device according to the first embodiment.

[0021] Figure 2 Figure 2 is a diagram for explaining a method of generating ground truth data for instance segmentation processing.

[0022] Figure 3 Figure 3 is a diagram showing an example of the hardware configuration of a learning device according to the first embodiment.

[0023] Figure 4 Figure 4 is a diagram showing an example of the functional configuration of a learning device according to the first embodiment.

[0024] Figure 5 Figure 5 is a diagram showing an example of an image data group.

[0025] Figure 6 Figure 6 is the first diagram showing an application example of a learning device according to the first embodiment.

[0026] Figure 7 Figure 7 is a diagram showing an example of input data and ground truth data of learning data.​​​​​​​​​​​​​​

[0027] Figure 8 Figure 8 Figure 2 shows an application example of the learning device according to the first embodiment.

[0028] Figure 9 Figure 9 Figure 13 is a diagram showing an example of input data and true value data for the first additional learning data.

[0029] Figure 10 Figure 10 Figure 3 shows an application example of the learning device according to the first embodiment.

[0030] Figure 11 Figure 11 Figure 27 is a diagram showing an example of input data and true value data for the Nth additional learning data.

[0031] Figure 12 Figure 12 Figure 34 is a diagram showing the result obtained by performing instance segmentation processing using the Nth-order learning completed segmentation model.

[0032] Figure 13 Figure 13 Figure 41 is a flowchart showing the flow of learning processing and learning data generation processing performed by the learning system.

[0033] Figure 14 Figure 14 Figure 48 is a diagram showing an example of the functional configuration of the learning device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0034] Hereinafter, each embodiment will be described with reference to the accompanying drawings. It should be noted that in this specification and the drawings, the same reference numerals are given to components having substantially the same functional configuration to omit redundant description.

[0035] [First Embodiment]

[0036] [System Configuration of Learning System]

[0037] First, the system configuration of the entire learning system including the learning device according to the first embodiment will be described. Figure 1 Figure 64 is a diagram showing an example of the system configuration of the learning system including the learning device according to the first embodiment. As Figure 1 shown, the learning system 100 includes a camera device 110, an image data acquisition device 120, an image data storage device 130, a learning device 140, and an image data processing device 150.

[0038] ​​​​​​​​​​​​​​The imaging device 110 captures an object and sends the captured image data to the image data acquisition device 120. As the imaging device 110, a digital camera, an optical microscope, a scanning electron microscope (SEM), a transmission electron microscope (TEM), etc. can be used according to the size of the object.

[0039] The image data acquisition device 120 divides the image data captured by the imaging device 110, classifies the divided multiple image data into multiple image data groups, and then stores them in the image data storage device 130. The number of divisions of the image data is adjusted according to the processing capabilities of the learning device 140 and the image data processing device 150, for example.

[0040] It should be noted that each image data included in the 0th image data group (referred to as the default image data group) among the multiple image data groups stored in the image data storage device 130 is sent to the image data processing device 150.

[0041] In response to each image data included in the default image data group being sent to the image data processing device 150, the learning device 140 obtains learning data from the image data processing device 150. The learning data includes each image data included in the default image data group and each ground truth data in the case where instance segmentation processing is performed on each of the image data. In addition, the learning device 140 performs a learning process using the learning data including the obtained image data and the obtained ground truth data, and generates a learned instance segmentation model. It should be noted that it can be configured to use an existing learned instance segmentation model obtained by performing a learning process using an appropriate learning image data group instead of generating a learned instance segmentation model using the default image data group for subsequent learning processes.

[0042] In addition, the learning device 140 reads out each image data included in the 1st image data group other than the default image data group among the multiple image data groups stored in the image data storage device 130. In addition, the learning device 140 sends each output image data output by inputting the read image data into the learned instance segmentation model to the image data processing device 150 as the 1st output image data group.

[0043] In addition, in response to the first output image data group being sent, the learning device 140 acquires the first additional learning data from the image data processing device 150. The first additional learning data includes each piece of image data included in the first image data group, and each processed output image data (each ground truth data) obtained by processing each output image data included in the first output image data group. In addition, the learning device 140 generates the first-order learning data by adding the first additional learning data to the learning data (it should be noted that the learning data generated based on the default image data group is also referred to as the zero-order learning data). Furthermore, the learning device 140 uses the first-order learning data to perform a learning process and generates the first-order learned instance segmentation model.

[0044] It should be noted that the learning device 140 repeats the above process N times (N is an integer greater than or equal to 2). Specifically, the learning device 140 reads out each piece of image data included in the x-th image data group (1 ≤ x ≤ N) among the multiple image data groups stored in the image data storage device 130, excluding the default image data group. The learning device 140 sends each output image data output by inputting the read-out image data into the (x - 1)-th learned instance segmentation model to the image data processing device 150 as the x-th output image data group.

[0045] In addition, in response to the x-th output image data group being sent, the learning device 140 acquires the x-th additional learning data from the image data processing device 150. The x-th additional learning data includes each piece of image data included in the x-th image data group, and each processed output image data (each ground truth data) obtained by processing each image data included in the x-th output image data group. In addition, the learning device 140 generates the x-th order learning data by adding the x-th additional learning data to the (x - 1)-th order learning data. Furthermore, the learning device 140 uses the x-th order learning data to perform a learning process and generates the x-th order learned instance segmentation model.

[0046] When the default image data group is sent, the image data processing device 150 generates each ground truth data in the case of performing instance segmentation processing on each piece of image data included in the image data group. In addition, the image data processing device 150 associates each generated ground truth data with each piece of image data and sends it to the learning device 140 as learning data.

[0047] In addition, when the image data processing device 150 is sent the first image data group and the first output image data group, it processes each output image data included in the first output image data group into true value data. In addition, the image data processing device 150 associates each output image data (each true value data) included in the processed first output image data group with each image data included in the first image data group, and sends it to the learning device 140 as the first additional learning data.

[0048] In addition, when the image data processing device 150 is sent the x-th image data group and the x-th output image data group, it processes each output image data included in the x-th output image data group into true value data. In addition, the image data processing device 150 associates each output image data (each true value data) included in the processed x-th output image data group with each image data included in the x-th image data group, and sends it to the learning device 140 as the x-th additional learning data.

[0049] <Method for generating true value data for instance segmentation processing>

[0050] Next, a general method for generating true value data for instance segmentation processing will be described. Figure 2 It is a diagram for explaining a method for generating true value data for instance segmentation processing.

[0051] In Figure 2 , the image data 210 is an example of the image data that is used as the input data for learning data in instance segmentation processing. In addition, the image data 220 is an example of the image data that is used as the true value data for learning data in instance segmentation processing. As Figure 2 shown, in order to generate the true value data of the learning data, it is necessary to assign class labels to all the pixels corresponding to the objects in the image data 210. In Figure 2 the case of the image data 210 shown, in order to assign class labels in a way that detects horses, it is necessary to appropriately distinguish the contours of the pixels corresponding to horses and assign class labels. Therefore, if a plurality of learning data for instance segmentation processing are to be prepared, it will take a lot of time and the workload of the operator will also be high.

[0052] In contrast, in the learning device 140 according to the first embodiment, as described above, it is configured to perform the learning process and the instance segmentation process in combination,

[0053] · For the default image data group, generate the true value data of the learning data from scratch. On the other hand,

[0054] · For image data groups after the first one, instead of generating the ground truth data of the learning data from scratch, the ground truth data of the learning data is generated by processing each output image data included in the output image data group.

[0055] Thereby, the workload of the operator in generating the learning data can be reduced.

[0056] <Hardware configuration of the learning device>

[0057] Next, the hardware configurations of the respective devices (image data acquisition device 120, learning device 140, image data processing device 150) included in the learning system 100 will be described. It should be noted that since the hardware configurations of the respective devices included in the learning system 100 have substantially the same hardware configuration, the hardware configuration of the learning device 140 will be described here.

[0058] Figure 3 is a diagram showing an example of the hardware configuration of the learning device according to the first embodiment. As Figure 3 shown, the learning device 140 includes a processor 301, a memory 302, an auxiliary storage device 303, an I / F (Interface) device 304, a communication device 305, and a driver device 306. It should be noted that the respective hardware components of the learning device 140 are interconnected via a bus 307.

[0059] The processor 301 has various arithmetic devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 301 reads various programs (such as a learning program, etc.) onto the memory 302 and executes them.

[0060] The memory 302 has main storage devices such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 301 and the memory 302 form a so-called computer, and by the processor 301 executing various programs read onto the memory 302, various functions are realized by this computer.

[0061] The auxiliary storage device 303 stores various programs and various data used when the processor 301 executes various programs.

[0062] The I / F device 304 is a connection device connected to an operation device 311 and a display device 312 which are examples of user interface devices. The communication device 305 is a communication device for communicating with an external device (not shown) via a network.

[0063] The drive device 306 is a device for placing the recording medium 313. The recording medium 313 mentioned here includes media that record information in optical, electrical, or magnetic ways, such as CD-ROMs, floppy disks, magneto-optical disks, etc. Additionally, the recording medium 313 may include semiconductor memories such as ROMs and flash memories that record information electrically.

[0064] It should be noted that various programs to be installed in the auxiliary storage device 303 are installed, for example, by setting the distributed recording medium 313 in the drive device 306 and having the drive device 306 read out the various programs recorded in the recording medium 313. Alternatively, various programs to be installed in the auxiliary storage device 303 can be installed by downloading from a network via the communication device 305.

[0065] <Functional Configuration of the Learning Device>

[0066] Next, the functional configuration of the learning device 140 will be described. Figure 4 This is a diagram showing an example of the functional configuration of the learning device according to the first embodiment. As described above, a learning program is installed in the learning device 140, and by executing this learning program, the learning device 140 functions as

[0067] · Image data acquisition unit 410,

[0068] · x-th order learning completion segmentation model 420,

[0069] · Collection unit 430,

[0070] · x-th order learning data acquisition unit 440,

[0071] · Learning unit 450,

[0072] · x-th order learning data generation unit 460,

[0073] · N-th order learning data generation unit 470,

[0074] functions.

[0075] When the x-th image data group is sent from the image data storage device 130, the image data acquisition unit 410 inputs each image data included in the x-th image data group into the x-th order learning completion segmentation model 420.

[0076] The x-th stage learned segmentation model 420 is a model based on R-CNN (Region Based Convolutional Neural Networks or Regions with CNN features), and is generated by the learning unit 450 described below through learning processing. Here, R-CNN is exemplary, and as the x-th stage learned segmentation model 420, models such as Mask R-CNN, YOLACT, SOLO, or derivative models of these models can be used.

[0077] The x-th stage learned segmentation model 420 performs instance segmentation processing on each image data included in the x-th image data group input by the image data acquisition unit 410.

[0078] In addition, the x-th stage learned segmentation model 420 outputs each output image data through instance segmentation processing and notifies it to the collection unit 430.

[0079] The collection unit 430 collects each output image data output from the x-th stage learned segmentation model and sends it to the image data processing device 150 as the x-th output image data group.

[0080] When the x-th stage learning data generation unit 460 receives from the image data processing device 150

[0081] · Each image data included in the default image data group,

[0082] · Each ground truth data in the case where instance segmentation processing has been performed on each image data as learning data, the received learning data is stored in the x-th stage learning data storage unit 480.

[0083] In addition, when the x-th stage learning data generation unit 460 receives from the image data processing device 150

[0084] · Each image data included in the x-th image data group,

[0085] · Each processed output image data (each ground truth data) obtained by processing each output image data included in the x-th output image data group

[0086] As the x-th additional learning data, each image data and each ground truth data are added to the (x - 1)-th stage learning data already stored in the x-th stage learning data storage unit 480 to generate the x-th stage learning data.

[0087] The x-th order learning data acquisition unit 440 reads out the learning data stored in the x-th order learning data storage unit 480 and inputs it to the learning unit 450. Additionally, whenever the x-th order learning data is stored in the x-th order learning data storage unit 480, the x-th order learning data acquisition unit 440 reads out the x-th order learning data and inputs it to the learning unit 450.

[0088] The learning unit 450 includes a segmentation model 451 and a comparison / modification unit 452, and inputs the learning data input by the x-th order learning data acquisition unit 440 or the "input data" of the x-th order learning data to the segmentation model 451. Additionally, the learning unit 450 inputs the learning data input by the x-th order learning data acquisition unit 440 or the "ground truth data" of the x-th order learning data to the comparison / modification unit 452.

[0089] The segmentation model 451 is a model based on R-CNN (Region Based Convolutional Neural Networks or Regions with CNN features). Here, R-CNN is exemplary, and as the x-th order learning completed segmentation model 420, models such as Mask R-CNN, YOLACT, SOLO, or derivative models of these models can also be used.

[0090] When each image data included in the learning data or the "input data" of the x-th order learning data is input, the segmentation model 451 performs instance segmentation processing and outputs each output image data.

[0091] The comparison / modification unit 452 updates the model parameters of the segmentation model 451 based on each output image data output from the segmentation model 451 and each ground truth data included in the learning data or the "ground truth data" of the x-th order learning data.

[0092] In this way, by the learning unit 450 performing learning processes using the learning data and the x-th order learning data respectively, a learned completed segmentation model or an x-th order learned completed segmentation model is generated. It should be noted that the learned completed segmentation model and the x-th order learned completed segmentation model generated in the learning unit 450 function as the x-th order learned completed segmentation model 420. For example, the learned completed segmentation model generated by the learning unit 450 functions as the 0-th order learned completed segmentation model. The 1st order learned completed segmentation model generated by the learning unit 450 functions as the 1st order learned completed segmentation model.

[0093] The N-th order learning data generation unit 470 reads out the N-th order learning data stored in the x-th order learning data storage unit 480 by adding the N-th additional learning data from the x-th order learning data storage unit 480, and stores it in the N-th order learning data storage unit 490.

[0094] <Specific example of the image data group>

[0095] Next, a specific example of the image data group generated by the image data acquisition device 120 will be described. Figure 5 is a diagram showing an example of the image data group. In Figure 5 , the symbol 510 represents the image data of one image captured by the imaging device 110 and acquired by the image data acquisition device 120.

[0096] In addition, the symbol 511 represents the image data obtained by magnifying and displaying a part of the image data shown by the symbol 510. As shown by the symbol 511, the object to be photographed in the present embodiment is a group of fine particles. By performing instance segmentation processing on the image data obtained by photographing the group of fine particles, each fine particle is identified, and normal fine particles (normal particles) and aggregated fine particles (aggregated particles) are determined. It should be noted that in the enlarged view shown by the symbol 511, the symbols 511a and 511b are aggregated particles, and the other fine particles are normal particles.

[0097] It should be noted that as the fine particles targeted in the present disclosure, as Figure 5 shown as an example, particles with a diameter in the micron range can be cited. The upper limit of the average particle diameter of the fine particles is 50 μm or less, preferably 30 μm or less, more preferably 20 μm or less. The lower limit of the average particle diameter of the fine particles is 1.0 μm or more, preferably 2.0 μm or more, more preferably 2.5 μm or more. As the material constituting the fine particles, it can be an inorganic material containing a metal or the like, or an organic material. On the surface of the fine particles, a functional layer for exhibiting a specific function can be formed.

[0098] In addition, in Figure 5 , the symbol 520 shows the state where the image data is divided into a plurality of image data by the image data acquisition device 120 and the divided plurality of image data are classified into a plurality of image data groups. Specifically, the symbol 520 shows the state where the image data shown by the symbol 510 is divided into 20 parts to generate 20 pieces of image data. In addition, the symbol 520 shows the state where 5 pieces of image data in the first row among the 20 pieces of image data are classified into the default image data group. In addition, the symbol 520 shows the state where 15 pieces of image data in the second to fourth rows among the 20 pieces of image data are classified into the first image data group.

[0099] It should be noted that, in this embodiment, similarly, by Figure 5 dividing the image data of the 2nd to 10th sheets not shown in

[0100] <Application Example (1) of Learning Device>

[0101] Next, an application example of the learning device 140 (until the generation of the learned segmentation model) will be described. Figure 6 Fig. 1 is a diagram showing an application example of the learning device according to the first embodiment. As Figure 6 shown, when the image data 601 included in the default image data group is sent to the image data processing device 150, in the image data processing device 150,

[0102] · normal image data with a normal class label assigned to the normal particles of the image data 601, and

[0103] · aggregated image data with an aggregated class label assigned to the aggregated particles of the image data 601 are generated.

[0104] In addition, the image data processing device 150 generates

[0105] · using the image data 601 as input data,

[0106] · learning data with the normal image data and the aggregated image data as ground truth data, and sends it to the learning device 140.

[0107] The learning data sent from the image data processing device 150 is acquired by the x-th order learning data generation unit 460 and stored in the x-th order learning data storage unit 480 as the learning data 602. It should be noted that, in Figure 6 this example, due to space limitations, only one set of input data and ground truth data is shown as the learning data 602, but the learning data 602 includes five sets of input data and ground truth data corresponding to the number of image data included in the default image data group.

[0108] The five sets of input data and ground truth data included in the learning data 602 are read out to the x-th order learning data acquisition unit 440, and learning processing is performed by sequentially inputting them to the learning unit 450 to generate a learned segmentation model.

[0109] Figure 7 Fig. 2 is a diagram showing an example of the input data and ground truth data of the learning data, and shows one set of input data and ground truth data of the learning data 602 shown in Figure 6 Fig. 1 in an enlarged manner. In Figure 7Among them, the image data 601 is the input data of the learning data 602, and the symbols 601a and 601b are aggregated particles. In addition, the normal image data 701 and the aggregated image data 702 are the true value data of the learning data 602.

[0110] <Application Example of Learning Device (2)>

[0111] Next, an application example of the learning device 140 (until generating the first-order learning completed segmentation model) will be described. Figure 8 It is the second figure showing the application example of the learning device according to the first embodiment. As Figure 8 shown, the image data acquisition unit 410 acquires the image data 801 included in the first image data group and inputs it to the x-th order learning completed segmentation model 420 that functions as the zero-th order learning completed segmentation model. As a result, the x-th order learning completed segmentation model 420 outputs the output image data 811 and 812.

[0112] It should be noted that in the Figure 8 example, due to space limitations, only the image data 801 among the respective image data included in the first image data group is shown, but for example, 15 image data are included in the first image data group. Therefore, in the collection unit 430, 15 groups of output image data are collected as the first output image data group and sent to the image data processing device 150.

[0113] In addition, as Figure 8 shown, after the image data 801 included in the first image data group is sent to the image data processing device 150, in the image data processing device 150,

[0114] · For the fine particles in the output image data 811 included in the first output image data group, in the case where the normal class label is not assigned to the pixels corresponding to the normal particles, the normal class label is assigned. In addition, in the case where the normal class label is assigned to the pixels corresponding to the fine particles other than the normal particles, the normal class label is deleted. Thus, the output image data 811 is processed into normal image data.

[0115] · For the fine particles in the output image data 812 included in the first output image data group, in the case where the aggregation class label is not assigned to the pixels corresponding to the aggregated particles, the aggregation class label is assigned. In addition, in the case where the aggregation class label is assigned to the pixels corresponding to the fine particles other than the aggregated particles, the aggregation class label is deleted. Thus, the output image data 812 is processed into aggregated image data.

[0116] In addition, the image data processing device 150 will

[0117] · Using the image data 801 as the input data,

[0118] · The first additional learning data with the normal image data and the aggregated image data obtained by processing the output image data 811 and 812 as the ground truth data is sent to the learning device 140.

[0119] The first additional learning data sent from the image data processing device 150 is acquired by the x-th order learning data generation unit 460 and added to the learning data 602, thereby generating the first order learning data 820 and storing it in the x-th order learning data storage unit 480. It should be noted that in Figure 8 the example, due to space limitations, only one set of input data and ground truth data are shown as the learning data and the first additional learning data respectively. However, the learning data includes 5 sets of input data and ground truth data, and the first additional learning data includes 15 sets of input data and ground truth data. In other words, the first order learning data 820 includes a total of 20 sets of input data and ground truth data.

[0120] The 20 sets of input data and ground truth data included in the first order learning data 820 are read out into the x-th order learning data acquisition unit 440 and sequentially input into the learning unit 450 for learning processing to generate the first order learned segmentation model.

[0121] Figure 9 FIG. is a diagram showing an example of the input data and the ground truth data of the first additional learning data. In Figure 9 it, the image data 801 is the input data of the first additional learning data. Additionally, in Figure 9 it, the normal image data 811' and the aggregated image data 812' are the ground truth data of the first additional learning data.

[0122] It should be noted that in Figure 9 the example, the normal image data 811' is generated by performing a process of assigning a normal class label to the tiny particles that do not have a normal class label for the pixels corresponding to normal particles in the output image data 811 included in the first output image data set. It should be noted that the tiny particles indicated by the black arrows in the output image data 811 refer to the tiny particles that do not have a normal class label for the pixels corresponding to normal particles. Additionally, in Figure 9 the case of the example, the aggregated image data 812' shows the appearance of directly using the output image data 812 without processing it.

[0123] In this way, when generating the first additional learning data, in the first embodiment, the output image data 811 and 812 are processed to obtain the normal image data 811' and the aggregated image data 812'. Thus, compared with the case of generating the normal image data 811' and the aggregated image data 812' from scratch based on the image data 801, the workload of the operator when generating the learning data can be reduced.

[0124] <Application Example of Learning Device (3)>

[0125] Next, an application example of the learning device 140 (until generating the N-th order learning completed segmentation model) will be described. Figure 10 This is the third figure showing the application example of the learning device according to the first embodiment. Note that, here, N = 2 is taken as an example for description.

[0126] As Figure 10 shown, the image data acquisition unit 410 acquires the image data 1001 included in the N-th image data group and inputs it to the x-th order learning completed segmentation model 420 that functions as the (N - 1)-th order learning completed segmentation model. Thus, the x-th order learning completed segmentation model 420 outputs the output image data 1011 and 1012.

[0127] Note that, in the Figure 10 example, due to space limitations, only the image data 1001 among the respective image data included in the N-th image data group is shown, but the N-th image data group includes, for example, 180 image data. Therefore, in the collection unit 430, 180 groups of output image data are collected as the N-th output image data group and sent to the image data processing device 150.

[0128] In addition, as Figure 10 shown, after the image data 1001 included in the N-th image data group is sent to the image data processing device 150, in the image data processing device 150,

[0129] · For the fine particles in the output image data 1011 included in the N-th output image data group, when the normal class label is not assigned to the pixels corresponding to the normal particles, the normal class label is assigned. In addition, when the normal class label is assigned to the pixels corresponding to the fine particles other than the normal particles, the normal class label is deleted. Thus, the output image data 1011 is processed into the normal image data.

[0130] · For the fine particles in the output image data 1012 included in the Nth output image data group, if the aggregation class label is not assigned to the pixels corresponding to the aggregated particles, the aggregation class label is assigned. In addition, if the aggregation class label is assigned to the pixels corresponding to the fine particles other than the aggregated particles, the aggregation class label is deleted. Thus, the output image data 1012 is processed into aggregation image data.

[0131] In addition, the image data processing device 150 sends

[0132] · Using the image data 1001 as the input data,

[0133] · The Nth additional learning data with the normal image data and the aggregation image data obtained by processing the output image data 1011 and 1012 as the ground truth data is sent to the learning device 140.

[0134] The Nth additional learning data sent from the image data processing device 150 is obtained by the xth-order learning data generation unit 460 and added to the (N - 1)th-order learning data to generate the Nth-order learning data 1020, which is stored in the xth-order learning data storage unit 480. It should be noted that, in Figure 10 the example, due to space limitations, only 2 sets of input data and ground truth data are shown as the (N - 1)th-order learning data, and only 1 set of input data and ground truth data are shown as the Nth additional learning data. However, the (N - 1)th-order learning data includes 20 sets of input data and ground truth data, and the Nth additional learning data includes 180 sets of input data and ground truth data. In other words, the Nth-order learning data 1020 includes a total of 200 sets of input data and ground truth data.

[0135] The 200 sets of input data and ground truth data included in the Nth-order learning data 1020 are read out to the xth-order learning data acquisition unit 440 and sequentially input to the learning unit 450 for learning processing to generate a learned segmentation model.

[0136] Figure 11 is a diagram showing an example of the input data and ground truth data of the Nth additional learning data. In Figure 11 the image data 1001 is the input data of the Nth additional learning data. In addition, in Figure 11 the normal image data 1011’ and the aggregation image data 1012’ are the ground truth data of the Nth additional learning data.

[0137] It should be noted that, in Figure 11In the example of , the normal image data 1011' is generated by processing the output image data 1011 included in the Nth output image data group to assign normal category labels to the tiny particles that are not assigned normal category labels to the pixels corresponding to the normal particles. It should be noted that in the output image data 1011, the tiny particles indicated by the black arrows are tiny particles that are not assigned normal category labels to the pixels corresponding to the normal particles. Figure 11 In the case of the example of , the aggregated image data 1012 ′ shows that the output image data 1012 included in the N-th output image data group is used directly without processing it.

[0138] In this way, when generating the Nth additional learning data, in the first embodiment, the normal image data 1011' and the aggregated image data 1012' are obtained by processing the output image data 1011 and 1012. Thus, compared with the case where the normal image data 1011' and the aggregated image data 1012' are generated from scratch based on the image data 1001, the operator's workload when generating the learning data can be reduced.

[0139] It should be noted that the amount of output image data to be processed when generating the (x+1)th additional learning data can be smaller than the amount of output image data to be processed when generating the xth additional learning data. The reason is that the amount of data used for learning processing increases, and the processing accuracy of the xth order learning completed instance segmentation model is improved.

[0140] <Result of instance segmentation processing>

[0141] Next, a description is given of a result of an instance segmentation process based on an Nth-order learned segmentation model generated by performing a learning process on the segmentation model 451 using the Nth-order learning data.

[0142] Figure 12 is a diagram showing the result of performing instance segmentation processing using the Nth order learning complete segmentation model. Figure 12 As shown, the processing accuracy is improved by using the Nth-order learning data for learning processing. Therefore, the segmentation model is completed according to the Nth-order learning.

[0143] ·It is possible to appropriately assign normal category labels to pixels corresponding to normal particles among the tiny particles contained in the input image data ( Figure 12 of green).

[0144] · It is possible to appropriately assign aggregation category labels to pixels corresponding to aggregated particles among the microparticles contained in the input image data ( Figure 12 of blue-purple).

[0145] <Flow of learning process and learning data generation process>

[0146] Next, the flow of the learning process and the learning data generation process performed by the learning system 100 will be described. Figure 13 It is a flowchart showing the flow of the learning process and the learning data generation process performed by the learning system.

[0147] In step S1301, the imaging device 110 captures the microparticle group as the object object and generates a plurality of image data.

[0148] In step S1302, the image data acquisition device 120 divides the image data, classifies it into a plurality of image data groups, and stores it in the image data storage device 130.

[0149] In step S1303, the image data processing device 150 acquires the default image data group sent from the image data storage device 130. In addition, the image data processing device 150 generates learning data with each image data included in the image data group as input data and the normal image data and the aggregated image data generated based on each image data included in the image data group as the true value data.

[0150] In step S1304, the learning device 140 generates a learned instance segmentation model by performing a learning process using the learning data.

[0151] In step S1305, the learning device 140 substitutes "1" for x.

[0152] In step S1306, the learning device 140 inputs the x-th image data group into the (x - 1)-th learned instance segmentation model and collects the x-th output image data group.

[0153] In step S1307, the image data processing device 150 processes each output image data included in the x-th output image data group to generate a processed x-th output image data group including the processed output image data. In addition, the image data processing device 150 generates the x-th additional learning data with each image data included in the x-th image data group as input data and the processed output image data included in the processed x-th output image data group as the true value data.

[0154] In step S1308, the learning device 140 adds the x-th additional learning data to the (x - 1)-th order learning data to generate the x-th order learning data.

[0155] In step S1309, the learning device 140 performs a learning process using the x-th order learning data to generate an x-th order learned instance segmentation model.

[0156] In step S1310, the learning device 140 increments x.

[0157] In step S1311, the learning device 140 determines whether x exceeds N. When it is determined in step S1311 that x does not exceed N (in the case of "No" in step S1311), the process returns to step S1306.

[0158] On the other hand, when it is determined in step S1311 that x exceeds N (in the case of "Yes" in step S1311), the process proceeds to step S1312.

[0159] In step S1312, the learning device 140 stores the N-th order learning data in the N-th order learning data storage unit 490.

[0160] <Summary>

[0161] As can be seen from the above description, the learning device 140 according to the first embodiment performs the following processes. · Perform a learning process using learning data including each image data included in the default image data group and each ground truth data in the case where instance segmentation processing is performed on each image data included in the default image data group. Thus, an (x - 1)-th order learned instance segmentation model is generated.

[0162] · Collect the x-th output image data group output by inputting the x-th image data group in a plurality of image data groups into the (x - 1)-th order learned instance segmentation model.

[0163] · Generate the x-th order learning data by obtaining the processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into ground truth data and adding it to the (x - 1)-th order learning data.

[0164] · Perform a learning process using the x-th order learning data to generate an x-th order learned instance segmentation model.

[0165] In this way, in the learning device 140 according to the first embodiment, it is configured to perform the learning process and the instance segmentation process in combination, and generate the ground truth data of the learning data by processing each output image data, instead of generating the ground truth data of the learning data from scratch.

[0166] Thus, according to the first embodiment, the workload of the operator in generating the learning data can be reduced.

[0167] In addition, in the learning device 140 according to the first embodiment, when performing learning processing using the x-th order learning data, instead of performing additional re-learning processing on the instance segmentation model that has completed learning at the (x - 1)-th order, the learning processing can also be started from scratch using the segmentation model prepared at the beginning each time.

[0168] By generating learning data in this way and generating the instance segmentation model that has completed learning at the N-th order, the learning device 140 according to the first embodiment can avoid the deviated learning processing caused by insufficient learning data.

[0169] [Second Embodiment]

[0170] In the above first embodiment, the image data acquisition device 120 and the image data processing device 150 are configured as devices separate from the learning device 140. However, the functions of the image data acquisition device 120 and the image data processing device 150 can also be implemented in the learning device 140.

[0171] Figure 14 FIG. is a diagram showing an example of the functional configuration of the learning device according to the second embodiment. The difference from Figure 4 the functional configuration of the learning device 140 according to the first embodiment shown is that in the case of the learning device 1400, it has a classification unit 1410 and an image data processing unit 1420.

[0172] The classification unit 1410 segments a plurality of image data captured by the imaging device 110 and acquired by the image data acquisition unit 410. In addition, the classification unit 1410 classifies the segmented plurality of image data into a plurality of groups of image data.

[0173] In addition, the classification unit 1410 notifies the default image data group among the plurality of groups of image data to the image data processing unit 1420. In addition, whenever the x-th order instance segmentation model 420 that has completed learning is updated, the classification unit 1410 notifies the x-th group of image data to the image data processing unit 1420 and inputs the x-th group of image data into the x-th order instance segmentation model 420 that has completed learning.

[0174] When notified of the default image data group by the classification unit 1410, the image data processing unit 1420 generates respective ground truth data in the case of performing instance segmentation processing on each image data included in the image data group. In addition, the image data processing unit 1420 associates the generated respective ground truth data with the respective image data and notifies it to the x-th order learning data generation unit 460 as learning data.

[0175] In addition, in the case where the x-th image data group is notified by the classification unit 1410 and the x-th output image data group is notified by the collection unit 430, the image data processing unit 1420 processes each output image data included in the x-th output image data group into true value data.

[0176] Furthermore, the image data processing unit 1420 associates each output image data (each true value data) included in the processed x-th output image data group with each image data included in the x-th image data group, and notifies the x-th stage learning data generation unit 460 as the x-th additional learning data.

[0177] In this way, by implementing the functions of the image data acquisition device 120 and the image data processing device 150 in the learning device 1400, according to the second embodiment, the same effects as those of the first embodiment can be obtained.

[0178] It should be noted that in the above description, it is assumed that both the function of the image data acquisition device 120 and the function of the image data processing device 150 are implemented in the learning device 1400. However, either the function of the image data acquisition device 120 or the function of the image data processing device 150 can also be implemented in the learning device 1400.

[0179] [Other Embodiments]

[0180] In the above embodiments, the details of the processing method when processing each output image data included in the x-th output image data group are not mentioned. However, when processing each output image data included in the x-th output image data group, for example, it can be configured to determine the fine particles to be processed by calculating the difference between each image data included in the x-th image data group, and then perform the processing.

[0181] In addition, in the above embodiments, the details of the classification method when classifying the divided multiple image data into multiple image data groups are not mentioned. However, when classifying the divided multiple image data into multiple image data groups, for example

[0182] · Classify in such a way that the number of image data included in the x-th image data group is larger than the number of image data included in the default image data group.

[0183] · Classify in such a way that the number of image data included in the (x + 1)-th image data group is larger than the number of image data included in the x-th image data group.

[0184] There is a relatively large number of fine particles for which class labels are to be assigned to pixels in order to generate ground truth data from each image data included in the default image data group. On the other hand, there is a relatively small number of fine particles for which class labels are to be assigned to pixels or class labels are to be deleted from pixels in order to process each output image data included in the x-th output image data group into each ground truth data.

[0185] In addition, in the case of each output image data included in the (x + 1)-th output image data group, the number of fine particles for which class labels are to be assigned to pixels or class labels are to be deleted from pixels in order to process them into each ground truth data is smaller than that of each output image data included in the x-th output image data group.

[0186] In other words, even if the number of image data in the x-th image data group is larger than the number of image data in the default image data group, the number of fine particles for which class labels are to be assigned to pixels or class labels are to be deleted from pixels will not increase, and the workload of the operator will not increase either. Similarly, even if the number of image data in the (x + 1)-th image data group is larger than the number of image data in the x-th image data group, the number of fine particles for which class labels are to be assigned to pixels or class labels are to be deleted from pixels does not increase, and the workload of the operator does not increase. Therefore, by adopting the above classification method, a large amount of ground truth data can be stored without increasing the workload of the operator.

[0187] In addition, in each of the above embodiments, although the object object is exemplified as fine particles with a diameter in the micron order, the size of the object object may also be in the sub-micron order or may be in the millimeter order.

[0188] In addition, in each of the above embodiments, although the segmentation model 451 is exemplified as a model that performs instance segmentation processing, it may also be a model that performs segmentation processing other than instance segmentation processing.

[0189] In addition, in each of the above embodiments, although an example of the use of the N-th order learned and completed segmentation model generated by the learning device 140 is not mentioned, the N-th order learned and completed segmentation model is assumed to be used for, for example, a process of determining normal particles and aggregated particles. At this time, the determination result based on the N-th order learned and completed segmentation model can be used to control the manufacturing conditions of the device for manufacturing fine particles.

[0190] It should be noted that the present invention is not limited to the configurations shown herein, such as combinations with other elements in the configurations exemplified in the above embodiments. Regarding these matters, changes can be made within the scope not departing from the gist of the present invention, and appropriate provisions can be made according to their application forms.

[0191] This application is based on Japanese Patent Application No. 2022-191298 filed on November 30, 2022, and the entire contents of this Japanese patent application are incorporated herein by reference.

[0192] Symbol Explanation

[0193] 100: Learning System

[0194] 110: Imaging Device

[0195] 120: Image Data Acquisition Device

[0196] 140: Learning Device

[0197] 150: Image Data Processing Device

[0198] 410: Image Data Acquisition Unit

[0199] 420: x-th Order Learning Completion Segmentation Model

[0200] 430: Collection Unit

[0201] 440: x-th Order Learning Data Acquisition Unit

[0202] 450: Learning Unit

[0203] 460: x-th Order Learning Data Generation Unit

[0204] 470: N-th Order Learning Data Generation Unit

[0205] 1400: Learning Device

[0206] 1410: Classification Unit

[0207] 1420: Image Data Processing Unit.

Claims

1. A learning device, comprising: A learning unit that performs learning processing using learning data to generate a learned completed model, where the learning data includes each image data included in a default image data group and each ground truth data in the case where each image data included in the default image data group is subjected to segmentation processing; A collection unit that collects an x-th output image data group output by inputting an x-th image data group among a plurality of image data groups into an (x - 1)-th learning completed model, where 1 ≤ x ≤ N, where N is an integer greater than or equal to 2; and A generation unit that generates the x-th order learning data by obtaining a processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into each ground truth data and adding it to the (x - 1)-th order learning data, wherein the learning unit performs learning processing using the x-th order learning data to generate the x-th order learned completed model.

2. The learning device according to claim 1, further comprising: A storage unit that stores the N-th order learning data generated by the generation unit.

3. The learning device according to claim 1 or 2, wherein The number of image data included in the x-th image data group is larger than the number of image data included in the default image data group.

4. The learning device according to any one of claims 1 to 3, wherein The number of image data included in the (x + 1)-th image data group is larger than the number of image data included in the x-th image data group.

5. The learning device according to any one of claims 1 to 4, wherein The ground truth data includes image data with a class label of normal fine particles assigned to pixels and image data with a class label of aggregated fine particles assigned to pixels.

6. A learning method, comprising the following steps executed by a computer of a learning device: A step of performing learning processing using learning data to generate a learned completed model, where the learning data includes each image data included in a default image data group and each ground truth data in the case where each image data included in the default image data group is subjected to segmentation processing; A step of collecting the x-th output image data group output by inputting the x-th image data group among a plurality of image data groups into the (x-1)-th stage learning completed model, where 1 ≤ x ≤ N, where N is an integer greater than or equal to 2; A step of generating the x-th order learning data by obtaining a processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into each ground truth data and adding it to the (x - 1)-th order learning data; and A step of performing learning processing using the x-th order learning data to generate the x-th order learned completed model.

7. A learning program that causes a computer of a learning device to execute the following steps: A step of performing learning processing using learning data to generate a learned completed model, where the learning data includes each image data included in a default image data group and each ground truth data in the case where each image data included in the default image data group is subjected to segmentation processing; A step of collecting the x-th output image data group output by inputting the x-th image data group among a plurality of image data groups into the (x - 1)-th learning completed model, where 1 ≤ x ≤ N, where N is an integer greater than or equal to 2; A step of generating the x-th order learning data by obtaining a processed x-th output image data group obtained by processing each output image data included in the collected x-th output image data group into each ground truth data and adding it to the (x - 1)-th order learning data; and A process of performing learning processing using the x-th order learning data to generate an x-th order learned model.

Images