Relearning support system, relearning support method, and relearning support program

Abstract

To support and facilitate work for relearning a learned model.SOLUTION: The storage unit stores one or a plurality of learned models learned using at least one microscopic image, one or a plurality of revisions indicating a version of the learned model associated with each of the learned models, and one or a plurality of pieces of design information for the learned model associated with the one or the plurality of revisions. The control device receives the selection of at least one learned model, acquires one or a plurality of revisions and one or a plurality of pieces of design information for the selected learned model from the storage unit, displays the name of the selected learned model in the first display area 810 of the model design information screen 800, and displays the acquired revision and the acquired design information in the second display area 820 of the model design information screen 800 in association with each other.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The disclosure of this specification relates to a relearning support system, a relearning support method, and a relearning support program. [Background technology]

[0002] For an image inspection device that uses a classifier obtained through machine learning to determine whether an object is good or bad from a captured image of the object, a technique is known in which the device performs re-learning using additional images that the user recognizes as good or bad product images (see, for example, Patent Document 1).

[0003] Furthermore, a technology called TMRNet is known as a technology for recognizing actions related to a task from a video of the task (see, for example, Non-Patent Document 1). TMRNet is an abbreviation for Temporal Memory Relation Network, and is a technology that identifies what task the action shown in the current frame is, based on the relationship between each of multiple frames. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 7287791 [Non-patent literature]

[0005] [Non-Patent Document 1] Yueming Jin, 5 others, “Temporal Memory Relation Network for Workflow Recognition from Surgical Video”, IEEE Transactions on Medical Imaging, Volume 40, Issue 7, July 2021 Summary of the Invention [Problem to be solved by the invention]

[0006] An inference model (trained model) generated by machine learning can be used to classify the tasks involved in videos of work processes on an object captured using a microscope. When this trained model is actually used, re-training may be repeatedly performed to obtain an updated version of the trained model, for example, whenever a change occurs in the work process or to meet a request for improved classification accuracy.

[0007] When relearning, the design information of each old version of the trained model (such as the time the model was created, information about the video that is the training data, the shooting conditions of the video, the performance of the model at the time of creation, and information about the worker who was the subject) is important for tracing the design intent and history of each old version of the model.

[0008] In view of the above, an object of one aspect of the present invention is to support the work of relearning a trained model and make the work easier. [Means for solving the problem]

[0009] A relearning support system according to one embodiment of the present invention includes a storage unit and a control unit. The storage unit stores one or more trained models, one or more pieces of version information associated with each of the trained models, and one or more pieces of design information for the trained models associated with the one or more pieces of version information. The one or more trained models are models trained using at least one microscope image. The control unit accepts selection of at least one trained model and acquires one or more pieces of version information and one or more pieces of design information for the selected trained model from the storage unit. The control unit then displays information about the selected trained model in a first display area of ​​a display, and associates the acquired version information with the acquired design information and displays them in a second display area of ​​the display. [Effects of the Invention]

[0010] According to the above aspect, it is possible to assist in the task of relearning a trained model, and to make the task easier to perform. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a flowchart showing an example of a procedure for creating an AI model. [Figure 2] 10 is a flowchart showing an example of a procedure for updating an AI model. [Figure 3] FIG. 1 is a diagram illustrating an example of a relearning support system. [Figure 4] FIG. 1 illustrates an example of a hardware configuration of a computer. [Figure 5] FIG. 2 is a diagram illustrating an example of a data structure of a model design information DB. [Figure 6] FIG. 10 is a diagram illustrating the number of classes classified into moving images. [Figure 7] FIG. 10 is a diagram illustrating an example of the data structure of a model-related file information DB. [Figure 8] 10 is a flowchart showing the processing contents of an example of a relearning support process. [Figure 9] FIG. 10 is a diagram showing an example of a model selection screen. [Figure 10] 10 is a flowchart showing processing details of an example of a model design information screen process; [Figure 11] FIG. 10 is a diagram showing a first example of a model design information screen. [Figure 12] 10 is a flowchart showing the processing contents of an example of a sorting process. [Figure 13] FIG. 10 is a diagram showing a second example of a model design information screen. [Figure 14] 10 is a flowchart showing an example of a moving image tag information screen process; [Figure 15] FIG. 10 is a diagram showing an example of a moving image tag information screen. [Figure 16] 10 is a flowchart showing processing details of an example of a model performance comparison screen process. [Figure 17]FIG. 10 is a diagram showing an example of a model performance comparison screen. [Figure 18] 10 is a flowchart showing an example of a score transition display screen process; [Figure 19] FIG. 10 is a diagram showing an example of a score transition display screen. [Figure 20] 10 is a flowchart showing the processing contents of an example of a re-learning process. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0013] Even today, with the increasing automation of work using robots and other tools, there are still many products that require manual assembly, and medical equipment is one example. Because the assembly of precision equipment like medical equipment involves many delicate tasks, it is often done under a microscope, and stereomicroscopes that allow the user to view the object in three dimensions with both eyes are often used for such work. Working under such a microscope is difficult, and variations in the work are likely to occur.

[0014] To reduce variations in work, the work may be limited to only trained workers. However, since variations in work due to individual proficiency are unavoidable, the work may be recorded under a microscope to enable confirmation of the appropriateness of the work and results. This recording method may involve capturing video of the work or results using a microscope camera.

[0015] The amount of video obtained in this way is large in daily product production. For this reason, it is not practical for a reviewer to check each video segment one by one to determine which task in a series of assembly processes each video segment represents. Therefore, a method has recently been proposed in which an AI model divides a video recording of a series of assembly processes and classifies them by task. "AI" stands for artificial intelligence. For example, the aforementioned TMRNet can be used as an AI model for this purpose.

[0016] Here, we will explain the process of creating an AI model that classifies each task in the product assembly process. Figure 1 is a flowchart showing an example of the procedure for creating an AI model.

[0017] In the AI ​​model creation process, first, the overall design of the AI ​​model to be created (e.g., how many video segments to divide the video into, which tasks each video segment should be classified into, etc.) is considered (S11). Next, the learning video showing the assembly process tasks is acquired (S12).

[0018] Next, for each video acquired in step S12, annotation is performed according to the results of the review in step S11 (S13). The annotation process involves marking the image frames at the boundary between two consecutive video segments when dividing the target video into multiple video segments.

[0019] Next, various conditions (learning conditions) for machine learning to create an AI model are set (S14), such as the number of learning iterations and the threshold for determining convergence of the learning.

[0020] Next, under the learning conditions set by the work in S14, machine learning and validation of the learning results are carried out using the annotated learning videos obtained by the work up to S13 as training data (S15).

[0021] Next, to test the AI ​​model obtained as a result of the learning process in S15, the AI ​​model is made to classify tasks shown in videos other than the training data (S16).Then, it is determined whether the results of this test are valid (S17).

[0022] If the test results are determined to be valid, the AI ​​model creation process ends. On the other hand, if the test results are determined to be invalid, the AI ​​model is recreated. Specifically, the process of re-acquiring the learning video (S18), re-annotating (S19), and re-setting the learning conditions (S20) is repeated through trial and error until the test results of S16 are determined to be valid.

[0023] An AI model is completed through this kind of creation process, for example.

[0024] After creating an AI model, it may become necessary to update the AI ​​model due to circumstances such as changes to the assembly process or a desire to improve the accuracy of task classification. Next, we will explain the process of updating an AI model. Figure 2 is a flowchart showing an example of the procedure for updating an AI model.

[0025] When an AI model needs to be updated, first, the design information at the time of creation of the current version of the AI ​​model to be updated and each version prior to the current version is referenced (S21). Next, based on the design information referenced in S21, the overall design of the updated version of the AI ​​model is reviewed (S22). Note that the design information includes, for example, the time of model creation, information about the video that serves as training data, the shooting conditions of the video, the model's performance at the time of creation, and information about the worker who was the subject of the model.

[0026] By referring to the design information through the S21 process, model developers updating AI models can understand the design intent and process behind the creation of the old version of the AI ​​model, and obtain an updated version of the AI ​​model that improves on problems in the old version that are derived from the design intent and process.

[0027] Next, based on the results of the review work in S22, a decision is made as to whether or not it is necessary to acquire additional new learning videos, which are different from those used in the machine learning to create the previous version of the AI ​​model (S23).

[0028] If it is determined that additional acquisition is necessary, the following steps are performed in sequence: additional acquisition of training videos (S24), annotation of the additionally acquired videos (S26), and resetting of the training conditions associated with the additional acquisition of videos (S28). After that, as a re-learning task, the machine learning and validation tasks of S15 in the AI ​​model creation task shown in Figure 1, and the subsequent tasks from S16 onwards are performed in sequence.

[0029] On the other hand, if it is determined that there is no need to acquire additional training videos, the next step is to determine whether or not changes are needed to the annotations attached to the already acquired training videos that were used to create the old version of the AI ​​model, based on the results of the review work in S22 (S25).

[0030] If it is determined that annotation changes are necessary, the acquired learning video is annotated (S26) for the changes, and the learning conditions are reset (S28) in response to the annotation changes. After that, as a re-learning process, the machine learning and validation processes of S15 in the AI ​​model creation process shown in Figure 1, and the subsequent processes from S16 onwards, are performed in sequence.

[0031] On the other hand, if it is determined that neither additional acquisition of learning videos nor changes to annotations is necessary, the next step is to determine whether or not the learning conditions need to be reset based on the results of the review work in S22 (S27).

[0032] If it is determined that the learning conditions need to be reset, the learning conditions are reset (S28). After that, as a re-learning task, the machine learning and validation tasks of S15 in the AI ​​model creation task shown in Figure 1, and the subsequent tasks from S16 onwards are performed in sequence.

[0033] On the other hand, if it is determined that there is no need to acquire additional learning videos, change annotations, or reset learning conditions, the review process in S22 is performed again, and then the process from S23 onwards is performed again.

[0034] When the AI ​​model update work is completed as described above, the data of the updated AI model is overwritten on the data of the AI ​​model before the update and saved. Note that instead of overwriting and saving, the data of the updated AI model may be saved as new data of a model separate from the data of the AI ​​model before the update.

[0035] For example, consider the case where an AI model with the model name "Trained Model 1" is updated. In this case, when the data is overwritten and saved, the model data is updated, while the setting information associated with the model name "Trained Model 1" is retained. In addition, the revision associated with the AI ​​model is updated.

[0036] On the other hand, in the above case, when the updated AI model data is saved as new data for another model, the model data reflecting the updates made to the "Trained Model 1" model is saved under a different name, such as "Trained Model 1'." In this case, the "Trained Model 1'" AI model inherits the configuration information and version information associated with the "Trained Model 1" AI model, but the version information is updated from "Trained Model 1." Doing this is useful, for example, when you want to prepare AI models for different work environments or system operating environments, or when you want to save the old model in a separate file as a backup.

[0037] For example, the above steps are performed when updating an AI model. In this procedure, the review step in S22 serves as the basis for subsequent work, and this review step uses the design information of the old version of each AI model referenced in S21. Therefore, if this design information can be easily obtained, the burden of updating an AI model, i.e., the work of retraining a trained AI model, can be reduced.

[0038] Therefore, in the following description, as an embodiment of the present invention, we will explain a system that displays design information for each trained AI model created in the past in association with information indicating each version of the AI ​​model, thereby assisting model developers in updating AI models.

[0039] First, a description will be given of Fig. 3. Fig. 3 shows the overall configuration of an example of a relearning support system 1.

[0040] The relearning support system 1 includes a microscope 100, a control device 200, a monitor 300, and a plurality of input devices 400 (a mouse 401, a keyboard 402, a foot switch 403, and a barcode reader 404).

[0041] Microscope 100 is a stereo microscope that allows for stereoscopic viewing of a sample. A user can observe an optical image formed by the microscope optical system on the object side of eyepiece 106 with both eyes via eyepiece 106, allowing for stereoscopic viewing of the object. Microscope 100 is suitable for applications such as precision equipment assembly work.

[0042] The microscope 100 is equipped with a zoom lens that can be operated with a zoom handle 130. By operating the zoom handle 130, the user can change the observation magnification while continuing to look through the eyepiece 106 and observe the object.

[0043] The microscope 100 includes a focusing handle 140. By operating the focusing handle 140, the user can change the distance between the object and the objective lens 101 and bring the object into focus.

[0044] The microscope 100 is equipped with an imaging device 112 that captures an image of an object and acquires a moving image of the object. An eyepiece tube 120 to which an eyepiece 106 is attached is a triplet tube, and the imaging device 112 is attached to the eyepiece tube 120. The imaging device 112 is provided with a two-dimensional image sensor. The image sensor is not particularly limited, but may be, for example, a CCD image sensor or a CMOS image sensor. The moving image acquired by the imaging device 112 is output to the control device 200. Alternatively, the moving image may be output directly to the monitor 300.

[0045] Light split from the optical path of an optical system (not shown) included in the microscope 100 by a beam splitter such as a half mirror enters the imaging device 112 via an imaging lens (not shown).

[0046] The microscope 100 includes a projector 113 that projects an auxiliary image onto an image plane where an imaging lens forms an optical image. The projector 113 is a device that projects and superimposes an auxiliary image onto the image plane in accordance with a command from the control device 200. More specifically, the projector 113 superimposes the auxiliary image onto the image plane based on auxiliary image data, which will be described later. The type of the projector 113 is not particularly limited. The projector 113 may be configured using, for example, a liquid crystal device or a digital mirror device.

[0047] The projector 113 is provided inside the eyepiece tube 120. Light from the projector 113 is guided to the optical path of the optical system of the microscope 100.

[0048] The eyepiece tube 120 is provided with an operation unit 121. By operating the operation unit 121, the user can switch the projector 113 on and off and instruct it to start or stop superimposing the auxiliary image on the image plane.

[0049] The control device 200 controls the microscope 100. The control device 200 generates the auxiliary image data described above and outputs it to the microscope 100 (projector 113).

[0050] The monitor 300 and the input device 400 are connected to the control device 200. The monitor 300 is, for example, a liquid crystal display or an organic EL display, and functions as a display in the relearning support system 1. Note that "EL" is an abbreviation for electro-luminescence.

[0051] 4 shows an example of the hardware configuration of a computer 200a for realizing the control device 200 in the above-described relearning support system 1. This computer 200a includes, as hardware, a processor 201, a memory 202, a storage device 203, a reading device 204, a communication interface 206, and an input / output interface 207. The processor 201, the memory 202, the storage device 203, the reading device 204, the communication interface 206, and the input / output interface 207 are connected to one another via, for example, a bus 208.

[0052] The processor 201 may be, for example, a single processor, a multiprocessor, or a multi-core processor. The processor 201 reads and executes programs stored in the storage device 203 to perform various control processes including the relearning support process described below, and provides a function as a control unit in the relearning support system 1.

[0053] The memory 202 may be, for example, a semiconductor memory, and may include a RAM area and a ROM area. Note that "RAM" is an abbreviation for Random Access Memory, and "ROM" is an abbreviation for Read Only Memory.

[0054] The storage device 203 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device, and provides the function of a storage unit in the re-learning support system 1. More specifically, the storage device 203 stores configuration data of one or more trained models trained using, for example, a moving image (at least one microscope image) captured by the imaging device 112 of the microscope 100. The storage device 203 also stores a model design information DB 500 and a model-related file information DB 600, which will be described later. Note that "DB" is an abbreviation for database.

[0055] The reader 204 accesses the removable storage medium 205 in accordance with, for example, an instruction from the processor 201. The removable storage medium 205 is realized by, for example, a semiconductor device, a medium for inputting and outputting information by magnetic action, or a medium for inputting and outputting information by optical action. An example of a semiconductor device is a USB (Universal Serial Bus) memory. An example of a medium for inputting and outputting information by magnetic action is a magnetic disk. An example of a medium for inputting and outputting information by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), or a Blu-ray (registered trademark) disk.

[0056] The communication interface 206 communicates with other devices (such as the microscope 100) according to instructions from the processor 201, for example. The input / output interface 207 is, for example, an interface between an input device 400 and an output device. The input device 400 is, for example, a device such as a mouse 401, a keyboard 402, or a foot switch 403 that receives instructions from a user. The output device is, for example, a monitor 300 and an audio device such as a speaker. Note that various operations such as a "click operation" described below are described as operations performed by the mouse 401 as an example, but are not limited to operations performed by the mouse 401 as long as they are designation operations using the input device 400.

[0057] The program executed by the processor 201 is provided to the computer in the following form, for example. (1) It is pre-installed in the storage device 203. (2) Provided by removable storage medium 205. (3) Provided from a server such as a program server.

[0058] Note that the hardware configuration of the computer 200a for realizing the control device 200 described with reference to FIG. 4 is an example, and the embodiment is not limited to this. For example, part of the above-described configuration may be deleted, or new configuration may be added. Furthermore, in another embodiment, for example, part or all of the functions of the control device 200 may be implemented as hardware. FPGA (Field Programmable Gate Array), SoC (System-on-a-Chip), ASIC (Application Specific Integrated Circuit), and PLD (Programmable Logic Device) are examples of hardware that can implement the control device 200.

[0059] Next, a description will be given of the model design information DB 500 stored in the storage device 203. Fig. 5 shows an example of the data structure of the model design information DB 500.

[0060] Each trained model stored in the storage device 203 is associated with version information indicating the version of the trained model. In the model design information DB 500 in Figure 5, for each trained model, one or more pieces of version information indicating the version of the trained model are associated with one or more pieces of design information about the trained model. Note that in Figure 5, "model name" is the name given to the AI ​​model (trained model), and "revision" is an example of version information.

[0061] The design information is information that includes at least one of time information, personal information, learning information, and character information. In FIG. 5, "Number of Videos," "Number of Classifications," "Score," and "Number of Inferences" are examples of learning information, and "Video Information" and "Updater" are examples that include personal information. Also, "Use (any word)" is an example of character information, and "Creation Date and Time" and "Last Update Date and Time" are examples of time information. In other words, in the model design information DB 500 in FIG. 5, these pieces of design information about the trained model are associated with "revisions" that identify the versions of the trained model.

[0062] The design information in FIG. 5 will be further explained.

[0063] "Number of videos" is information about the number of training videos used in the machine learning performed when creating the trained model.

[0064] The "number of classifications" is information about the number of classifications that the trained model makes when classifying a single video obtained by photographing an assembly process with the microscope 100 into video segments of several tasks that make up the assembly process.

[0065] For example, in the video of the assembly process of a certain part shown in Figure 6, one video is classified into video segments from "Class 00" to "Class 06" for each task that makes up the assembly process. Therefore, in this example, the "number of classes" is "7".

[0066] Returning to the explanation of FIG. 5, the "score" is information about a value that quantitatively evaluates the trained model and represents the reliability of the classification of the assembly process video into video segments of each task performed by the trained model. In this embodiment, the "score" is calculated based on the convergence value of a loss function calculated during machine learning when creating the trained model. Note that values ​​calculated by other methods may also be used as the "score."

[0067] "Number of inferences" is information about the number of inferences performed using the trained model, i.e., the number of times the trained model was actually used to classify the assembly process video into video segments for each task.

[0068] "Video tag information" is tag information that was attached to the data of the training videos used for machine learning when creating the trained model. The training videos of the assembly process stored in the storage device 203 are attached with, for example, the name of the worker who performed the assembly process and information about the worker's dominant hand, as well as observation information such as the configuration and observation magnification of the microscope 100 used to capture the video. "Video tag information" shows all of this tag information that was attached to each of the training videos used for machine learning.

[0069] "Purpose ((any word))" is text information that expresses information about the creation of the trained model, such as the design intent and history of the trained model, and is information entered by the model developer who created or updated the trained model.

[0070] "Creation date and time" is information about the date and time when the trained model was created or updated.

[0071] The "last update date and time" is information on the date and time when the design information for the trained model was updated.

[0072] "Updater" is information about the name of the model developer who created or updated the trained model.

[0073] Next, a description will be given of the model-related file information DB stored in the storage device 203. Fig. 7 shows an example of the data structure of the model-related file information DB 600.

[0074] As described above, configuration data of trained models is stored in the storage device 203. Various data files related to the trained models are also stored in this storage device 203. The model-related file information DB 600 is used to manage the association between these data files and trained models.

[0075] In the model-related file information DB600 illustrated in Figure 7, for each trained model, a "revision" as version information indicating the version of the trained model is associated with the name of the data file stored in the storage device 203.

[0076] "Video file name" is information about the file name of the video data file of the training video used in the machine learning performed when creating the trained model.

[0077] "Annotation file name" is information about the file name of the annotation file for the training video indicated by "Video file name" that was used in the machine learning performed when creating the trained model. Note that an annotation file is a data file that stores information about annotations, such as information indicating the position in the training video of the annotations that were added to the training video during the machine learning.

[0078] "Loss function value data file name" is information about the file name of the loss function value data file for the machine learning performed when the trained model was created. The loss function value data file is a file that stores data that associates the number of iterations of learning in the machine learning performed when the trained model was created with the loss function value for each iteration of learning. The loss function value data file is used to display the model performance comparison screen described below.

[0079] In the model-related file information DB 600 illustrated in FIG. 7, annotation files are managed for each revision of the trained model. Alternatively, annotation files may be managed for each video file. That is, video files and annotation files may be associated one-to-one, and annotation information for the video file may be managed for each revision of the trained model within the corresponding annotation file. Furthermore, annotation information for each revision of the trained model for a video file may be embedded in the video file, and annotation management for each revision of the trained model may be performed within the video file.

[0080] Next, various processes performed by the processor 201 will be described.

[0081] [1: Re-learning support processing]

[0082] First, the relearning support process will be described with reference to a flowchart of FIG.

[0083] This relearning support process starts when the processor 201 acquires a process start instruction issued by a model developer who performs an update operation on a trained model, by operating the input device 400. When the process starts, first, in S101, a process is performed in which the model selection screen 700 illustrated in Fig. 9 is displayed on the monitor 300 connected to the input / output interface 207.

[0084] Here, a description will be given of the model selection screen 700 in Fig. 9. On the model selection screen 700, the information of "model name", "date", "dataset", and "AI model" is associated with each other.

[0085] "Model name" is the name of the AI ​​model whose configuration data is stored in the storage device 203, and "date" is the creation date of the AI ​​model. This information is obtained from the model design information DB 500 described above and displayed on the model selection screen 700.

[0086] "Dataset" represents the number of training videos planned to be used for machine learning when creating the AI ​​model, and the number of training videos actually used for that machine learning. If the numbers on both sides of the diagonal line in "Dataset" are the same, it means that all of the planned training videos were used for machine learning.

[0087] Furthermore, "AI model" indicates the creation status of the trained model, and "created" indicates that creation of the AI ​​model has already been completed. In the example of Figure 9, all "AI models" are "created," so creation work has been completed for all AI models whose model names are displayed on the model selection screen 700 in Figure 9.

[0088] The processor 201 generates the information displayed on the model selection screen 700 using the information shown in the model design information DB 500 and the model-related file information DB 600.

[0089] Returning to the explanation of Fig. 8, once the model selection screen 700 is displayed on the monitor 300, a process of acquiring an instruction operation for the input device 400 is then performed in S102. Then, a process of determining whether the acquired instruction operation is an operation for selecting the model name of an AI model is performed in S103.

[0090] 9, model selection screen 700 is arranged with model selection buttons 710, each showing the name of an AI model as a "model name." The model selection buttons 710 are icon buttons, and a click on the model selection button 710 is detected as an operation to select the model name of an AI model.

[0091] If it is determined in the determination process of S103 that the acquired operation is for selecting a model name, a model design information screen process is performed in S104. The model design information screen process is a process for switching the display screen on the monitor 300 from the model selection screen 700 to a model design information screen 800, which will be described later. The details of this process will be described later.

[0092] Thereafter, when the model design information screen processing is completed, the process returns to S101, and the process of displaying the model selection screen 700 is performed again.

[0093] On the other hand, if it is determined in the determination process of S103 that the acquired instruction operation was not a model name selection operation, a process is performed in S105 to determine whether the instruction operation acquired in S102 was a dataset selection operation.

[0094] In the model selection screen 700 illustrated in FIG. 9, a mouse pointer 720 points to the display position of the dataset for "Trained Model 1," and the movement of the mouse pointer 720 to this position is detected as an operation to select a dataset.

[0095] If it is determined in the determination process of S105 that the acquired instruction operation is an operation for selecting a data set, then in S106, a process is performed in which a moving image list screen 730 is popped up on the monitor 300 that is displaying the model selection screen 700. On the other hand, if it is determined in the determination process of S105 that the acquired instruction operation is not an operation for selecting a data set, then the process returns to S102, and the process of acquiring an instruction operation is performed again.

[0096] The video list screen 730 is a screen that displays a list of information about the training videos used for machine learning when generating the AI ​​model identified by the model name corresponding to the selected dataset. In the example of Figure 9, this information includes the creator's name ("ID"), creation date ("Date"), annotation work revision ("Rev"), and number of classifications ("Number of Classes") for the training video. This information is included in the tag information attached to the training video or the annotation file for the training video.

[0097] Following the process of S106, in S107, a process is performed to determine whether the selection operation of the data set acquired in the process of S102 has been completed. This determination process is repeated until it is determined that the selection operation has been completed, and when it is determined that the selection operation has been completed, the process returns to S101, where the pop-up display of the video list screen 730 is terminated and the process of displaying the model selection screen 700 is performed again.

[0098] [2: Model design information screen processing]

[0099] Next, the model design information screen processing will be described. The model design information screen processing is processing that is performed as processing of S104 when it is determined that the processor 201 has accepted the selection operation of the AI ​​model (trained model) in the determination processing of S103 of the re-learning support processing in Fig. 8. Fig. 10 is a flowchart showing the processing contents of an example of the model design information screen processing.

[0100] When the processing of FIG. 10 starts, first, in S111, a process is performed to acquire, from the storage device 203, various pieces of information about the trained model selected by operation in the determination process of S103 of the re-learning support process.

[0101] By the processing of S111, one or more pieces of version information about the selected trained model and design information corresponding to each piece of version information are acquired from the model design information DB 500. Also, model-related information about the selected trained model is acquired from the model-related file information DB 600. Furthermore, video (learning video) data, annotation data, and loss function values ​​identified by the file names indicated in the acquired model-related information are acquired from the storage device 203.

[0102] Next, in S112, a process is performed in which the model design information screen 800 shown in FIG. 11 is created and displayed on the monitor 300 using the various information acquired in the process of S111.

[0103] Here, a first example of the model design information screen 800 shown in FIG. 11 will be described.

[0104] The model design information screen 800 has a first display area 810, a second display area 820, a moving image selection area 830, and a moving image display area 840.

[0105] Through the processing of S112, processor 201 displays the name of the selected trained model as information about the model in first display area 810. Furthermore, through this processing, processor 201 displays design information about the selected trained model (the aforementioned time information, person information, learning information, and character information) in second display area 820, each associated with "revision" information, which is version information.

[0106] The design information displayed in the first display area 810 and the second display area 820 is information obtained from the model design information DB 500. By referring to this design information, a model developer who updates a trained model can easily understand the design intent and history at the time of model creation, and in particular, can appropriately recognize differences between revisions in the design intent and history.

[0107] The design information of the trained model for each revision displayed in the second display area 820 is displayed in a manner in which the design information for at least one revision is selected (in the example of FIG. 11, the design information is displayed in black text on a white background). By executing the process of S112, the processor 201 displays a list of each training video used in training to create the trained model of the selected revision, which was acquired by the process of S111, in the video selection area 830. Here, when a click operation is performed on the design information of an unselected revision in the second display area 820, the selection of the revision is changed, and the processor 201 displays a list of the training videos for the newly selected revision in the video selection area 830.

[0108] Furthermore, in the list display of learning videos in video selection area 830, one of the learning videos displayed is displayed in a selected state (in the example of FIG. 11, the video file name is displayed in black text on a white background). Processor 201 displays a video of the learning video displayed in this selected state in video display area 840. Note that the selection of a learning video in the list display in video selection area 830 can also be changed by clicking on an unselected learning video in the list display, and processor 201 displays a video of the newly selected learning video in video display area 840.

[0109] The model design information screen 800 further has icon buttons, namely a sort button 851, a tag information button 852, a performance comparison button 853, and a score transition button 854, as well as a "back" button 860 and a "create AI model" button 870. When any of these buttons is clicked, the display on the monitor 300 switches from the model design information screen 800 to one of the various screens associated with the clicked button.

[0110] Returning to the explanation of Fig. 10, once the model design information screen 800 is displayed on the monitor 300 by the process of S112, a process of acquiring an instruction operation for the input device 400 is then performed in S113. Then, a process of determining whether the acquired instruction operation is a click operation on any of the above-mentioned icon buttons is performed in S114.

[0111] If it is determined in the determination process of S114 that the acquired instruction operation is a click operation on one of the icon buttons, then in S115, each process associated with the clicked icon button is executed. The details of these processes will be described later.

[0112] After that, when the processing of S115 is completed, the processing returns to S111, and the processing of obtaining information about the selected trained model and displaying the model design information screen 800 by the processing of S112 is performed again.

[0113] On the other hand, if it is determined in the determination process of S114 that the acquired instruction operation was not a click operation on any of the icon buttons, the process proceeds to S116. Then, in S116, a process is performed to determine whether the instruction operation acquired in the process of S113 was a click operation on the "back" button 860. If it is determined in this determination process that the instruction operation was a click operation on the "back" button 860, this model design information screen process is terminated, and the process returns to the re-learning support process of FIG. 8.

[0114] On the other hand, if it is determined in the determination process of S116 that the instruction operation is not a click operation on the "back" button 860, the process returns to S113, and a process is performed to newly obtain an instruction operation on the input device 400. At this time, in response to an operation instruction on the model design information screen 800, the processor 201 may execute another process in accordance with the operation instruction.

[0115] The above processing is the model design information screen processing.

[0116] In the following explanation, various processes performed as the process of S115 in the model design information screen process will be explained.

[0117] [3: Sorting]

[0118] First, the sorting process will be described with reference to the flowchart of FIG.

[0119] This sorting process is executed by the processor 201 as the process of S115 when the instruction operation acquired by the process of S113 in the model design information screen process is a click operation on the sort button 851. This process is a process of rearranging the display order of the design information for each revision of the trained model in the second display area 820 of the model design information screen 800 in descending order of the "number of inferences."

[0120] As described above, the number of inferences is information about the number of times a trained model is actually used to classify a video of an assembly process into video segments for each task. Therefore, a trained model with a high number of inferences is presumed to be a model with high suitability for the design intent of the model and high performance. Therefore, rearranging the display in this way makes it possible for a model developer using the re-learning support system 1 to easily understand the relative high / low relationship between the performance levels of each revision of a trained model.

[0121] 12 starts, first, in S121, a process is performed in which the design information displayed for each revision in the second display area 820 on the model design information screen 800 is sorted in descending order of "number of inferences." Then, in the following S122, a process is performed in which the design information for each revision is displayed in the second display area 820 on the model design information screen 800 in the order after sorting.

[0122] Fig. 13 shows a second example of the model design information screen 800. In this second example, the display order of the design information for each revision displayed in the second display area 820 in the first example shown in Fig. 11 is sorted according to the "number of inferences." In the first example shown in Fig. 11, the design information for each revision is sorted in descending order of creation date (earliest revision), whereas in this second example, the design information for each revision is sorted in descending order of the "number of inferences."

[0123] 12, once the design information for each revision is displayed in the sorted order by the process of S122, a process of acquiring an instruction operation on the input device 400 is performed in S123. Then, a process of determining whether the acquired instruction operation is a click operation on any of the icon buttons described above is performed in S124.

[0124] If it is determined in the determination process of S124 that the acquired instruction operation is a click operation on one of the icon buttons, the process associated with the clicked icon button is executed in S125. The details of these processes will be described later.

[0125] Thereafter, when the process of S125 is completed, the process returns to S122, and the process of displaying the model design information screen 800 in which the design information for each revision is displayed in descending order of the "number of inferences" is performed again.

[0126] On the other hand, if it is determined in the determination process of S124 that the acquired instruction operation was not a click operation on any of the icon buttons, the process proceeds to S126. Then, in S126, a process is performed to determine whether the instruction operation acquired in the process of S123 was a click operation on the "back" button 860. If it is determined in this determination process that the instruction operation was a click operation on the "back" button 860, this sorting process is terminated, and the process proceeds to the re-learning support process of Fig. 8, where the process of S101, that is, the process of displaying the model selection screen 700, is performed.

[0127] On the other hand, if it is determined in the determination process of S126 that the instruction operation is not a click operation on the "back" button 860, the process returns to S123, and the instruction operation on the input device 400 is acquired again.

[0128] The above processing is the sorting processing.

[0129] In the sorting process illustrated in FIG. 12, the display order of the design information for each revision is sorted according to the number of inferences. Alternatively, this display order may be sorted based on other design information. That is, for example, the display order of the design information for each revision may be sorted in descending order of the length of time the trained model for each revision has been in use. Note that the length of time the trained model for a certain revision has been in use is, for example, the period from the creation date of the trained model for that revision to the creation date of the trained model for the next revision following that revision.

[0130] [4: Video tag information screen processing]

[0131] Next, the moving image tag information screen process will be described with reference to Fig. 14, which is a flowchart showing an example of the moving image tag information screen process.

[0132] The moving image tag information screen processing is executed by the processor 201 as processing of S115 when the instruction operation acquired by the processing of S113 in the model design information screen processing of Fig. 10 is a click operation on the tag information button 852. This processing is also executed by the processor 201 as processing of S125 when the instruction operation acquired by the processing of S123 in the sorting processing of Fig. 12 is a click operation on the tag information button 852.

[0133] When the processing of FIG. 14 starts, first, in S131, processing is performed to display the moving image tag information screen 900 shown in FIG. 15 on the monitor 300.

[0134] Here, the moving image tag information screen 900 shown in FIG. 15 will be described.

[0135] The video tag information screen 900 is a screen that displays, for each training video used in machine learning when creating a trained model, the tag information attached to the training video data in association with the training video. By referring to this video tag information screen 900, a model developer who updates a trained model can easily understand the situation when the training videos were acquired.

[0136] The moving image tag information screen 900 has a design information display area 910 , a tag information list display area 920 , and a tag information selection area 930 .

[0137] The design information display area 910 is an area where design information about a trained model is displayed. When the sort button 851 is clicked on the model design information screen 800 in FIG. 11 or 13, the design information about the revision selected in the second display area 820 is displayed in the design information display area 910.

[0138] The tag information list display area 920 is an area that displays a list of the learning videos used for learning when creating the trained model of the revision whose design information is displayed in the design information display area 910, in association with the tag information attached to each piece of data in the learning videos. This tag information list display area 920 is an example of a third display area.

[0139] The tag information selection area 930 is an area for individually selecting tag information from the design information of the trained model of the revision whose design information is displayed in the design information display area 910.

[0140] In the example of FIG. 15, five items of tag information are displayed in the design information display area 910: "Worker XX," "Right-handed," "Zoom 2X," "Worker YY," and "Left-handed." These are tag information items that were attached to any of the training videos used for training when creating the trained model of the revision whose design information is displayed in the design information display area 910. These five items are displayed in the tag information selection area 930.

[0141] When any of these items displayed in the tag information selection area 930 is clicked, the item clicked on is displayed in inverted display mode (white characters representing the item are displayed on a black background). At this time, the tag information of the same item displayed in association with the learning video in the tag information list display area 920 also changes to inverted display mode.

[0142] In the example of Fig. 15, three of the five items displayed in the tag information selection area 930, "Worker XX," "Right-handed," and "Zoom 2X," are displayed in inverted mode, indicating that these three items have been selected. Fig. 15 also shows that, as a result of this selection, the tag information of "Worker XX," "Right-handed," and "Zoom 2X," which are displayed in association with the learning video in the tag information list display area 920, are now displayed in inverted mode.

[0143] As described above, the video tag information screen 900 displays the selected tag information in association with the learning videos related to that tag information in response to the selection of tag information received in the tag information selection area 930. By displaying such a video tag information screen 900 on the monitor 300, a model developer who is updating a trained model can be provided with information to help them select learning videos to use in re-training to update the trained model.

[0144] Returning to the explanation of FIG. 14 , in the processing of S131, first, design information for the trained model of the revision selected in the second display area 820 when the sort button 851 on the model design information screen 800 is clicked is acquired. Then, the display of the design information display area 910 is created using the acquired design information. At this time, the model-related file information DB 600 is referenced to identify the video files of each training video used in training when creating the trained model of the revision selected in the second display area 820. Then, tag information is acquired from the video files, and the display of the tag information list display area 920 is created by associating the acquired tag information with the training video for each training video. Furthermore, the display of the tag information selection area 930 is created using the tag information included in the design information displayed in the design information display area 910. The processor 201 displays the video tag information screen 900, in which the display of each area has been created in this manner, on the monitor 300.

[0145] Next, in S132, a process is performed to acquire an instruction operation on the input device 400. Then, in S133, a process is performed to determine whether the acquired instruction operation is a click operation on any of the tag information displayed in the tag information selection area 930.

[0146] If it is determined in the process of S133 that the instruction operation is a click operation on tag information, then in S134, the same tag information as that clicked is displayed in reverse video in the tag information list display area 920 and the tag information selection area 930. Thereafter, the process returns to S132, and the process of acquiring the instruction operation on the input device 400 continues.

[0147] On the other hand, if it is determined in the process of S133 that the instruction operation was not a click operation on tag information, then in S135, a process is performed to determine whether or not the instruction operation was a click operation on the "Back" button 860 of the video tag information screen 900. If it is determined in this determination process that the instruction operation was a click operation on the "Back" button 860, then the video tag information screen process is terminated, and the process returns to the original process, that is, the model design information screen process or the sort process.

[0148] On the other hand, if it is determined in the determination process of S135 that the instruction operation is not a click operation on the "back" button 860, the process returns to S132, and the instruction operation on the input device 400 is acquired again.

[0149] [5: Model performance comparison screen processing]

[0150] Next, the model performance comparison screen processing will be described with reference to a flowchart of FIG.

[0151] The model performance comparison screen processing is executed by the processor 201 as processing of S115 when the instruction operation acquired by the processing of S113 in the model design information screen processing of Fig. 10 is a click operation on the performance comparison button 853. This processing is also executed by the processor 201 as processing of S125 when the instruction operation acquired by the processing of S123 in the sorting processing of Fig. 12 is a click operation on the performance comparison button 853.

[0152] When the processing of FIG. 16 starts, first, in S141, a process is performed to read and acquire from the storage device 203 the loss function value data file associated with the revision of the trained model selected in the second display area 820 of the model design information screen 800.

[0153] In the processing of S141, first, information on the revision indicating the version of the trained model selected in the second display area 820 when the performance comparison button 853 was clicked on the model design information screen 800 is obtained. At this time, if multiple revisions have been selected in the second display area 820, information on all selected revisions is obtained. Next, the model-related file information DB 600 is referenced to obtain the loss function value data file names corresponding to the obtained revisions, and the loss function value data files identified by the obtained file names are read from the storage device 203.

[0154] Next, in S142, a process is performed to display on the monitor 300 a model performance comparison screen 1000 illustrated in FIG. 17, which includes a graph of the loss function value data created using the data included in the loss function value data file obtained by the process of S141.

[0155] Here, the model performance comparison screen 1000 shown in Fig. 17 will be described. The model performance comparison screen 1000 is a screen that displays information indicating the performance of a trained model for each revision. A model developer who performs update work on a trained model can easily compare the performance of the trained model for each revision by referring to the model performance comparison screen 1000.

[0156] The model performance comparison screen 1000 has a performance comparison display area 1010. The performance comparison display area 1010 superimposes and displays graphs of loss function value data for each trained model for each revision as information indicating the performance of the trained model for each revision.

[0157] In the example of FIG. 17 , graphs of loss function value data for the trained models of three revisions, "1.0," "1.1," and "1.2," selected in the second display area 820 of the model design information screen 800, are displayed in the performance comparison display area 1010. The graphs of loss function value data are graphs that represent the relationship between the number of learning iterations in machine learning when generating the trained model and the loss function value. This display mode in which graphs of loss function value data for multiple revisions are superimposed is an example of a display mode that allows comparison of information indicating the performance of the trained model for each of multiple revisions.

[0158] In the example of FIG. 17, the three graphs are distinguished by different line types, but instead, for example, the three graphs may be distinguished by different line colors.

[0159] When the input device 400 is operated while this model performance comparison screen 1000 is displayed, and the mouse pointer 720 points to a graph of loss function value data in the performance comparison display area 1010 or the legend display for that graph, a design information display area 910 appears. The design information display area 910 is an area that displays information about the trained model of the revision whose loss function value is indicated in the graph or legend display pointed to by the mouse pointer 720. The example in FIG. 17 shows a state in which the mouse pointer 720 points to the legend display for revision "1.2," causing design information about the trained model of revision "1.2" to be displayed in the design information display area 910.

[0160] 16, in the process of S142, a graph of loss function value data for each revision is created from the data contained in each of the loss function value data files read in the process of S141. Then, the graphs are superimposed to create the display of the performance comparison display area 1010. The processor 201 causes the monitor 300 to display the model performance comparison screen 1000 on which the display of the performance comparison display area 1010 has been created in this way.

[0161] Next, in S143, a process is performed to acquire an instruction operation on the input device 400. Then, in S144, a process is performed to determine whether the acquired instruction operation is an operation to select a revision of a trained model by selecting a graph or legend display in the performance comparison display area 1010.

[0162] If it is determined in the determination process of S144 that the instruction operation is an operation to select a revision, the process proceeds to S145. Then, in S145, the design information display area 910 is popped up, and design information of the trained model of the selected revision is acquired from the model design information DB 500 and displayed in the design information display area 910. Thereafter, the process returns to S143, and the process of acquiring the instruction operation is performed again.

[0163] On the other hand, if it is determined in the determination process of S144 that the instruction operation was not an operation to select a revision, the process proceeds to S146. Then, in S146, a process is performed to determine whether the acquired instruction operation was an operation to deselect a revision by moving the mouse pointer 720 from the performance comparison display area 1010.

[0164] If it is determined in the determination process of S146 that the instruction operation is an operation to cancel the selection of a revision, the process proceeds to S147. Then, in S147, the pop-up display of the design information display area 910 is cleared from the model performance comparison screen 1000, and the screen returns to the original state. Thereafter, the process returns to S143, and the process of acquiring the instruction operation is performed again.

[0165] On the other hand, if it is determined in the determination process of S146 that the instruction operation was not an operation to cancel the selection of a revision, the process proceeds to S148. Then, in S148, a process is performed to determine whether the acquired instruction operation was a click operation on the "Back" button 860 of the model performance comparison screen 1000. If it is determined in this determination process that the instruction operation was a click operation on the "Back" button 860, the model performance comparison screen process is terminated, and the process returns to the original process, that is, the model design information screen process or the sort process.

[0166] On the other hand, if it is determined in the determination process of S148 that the instruction operation is not a click operation on the "back" button 860, the process returns to S143, and the instruction operation on the input device 400 is acquired again.

[0167] [6: Score transition display screen processing]

[0168] Next, the score transition display screen process will be described with reference to the flowchart of FIG.

[0169] The score transition display screen processing is executed by the processor 201 as processing of S115 when the instruction operation acquired by the processing of S113 in the model design information screen processing of Fig. 10 is a click operation on the score transition button 854. This processing is also executed by the processor 201 as processing of S125 when the instruction operation acquired by the processing of S123 in the sorting processing of Fig. 12 is a click operation on the score transition button 854.

[0170] When the processing of Figure 18 starts, first, in S151, the score values ​​of each revision for the trained model whose design information is displayed on the currently displayed model design information screen 800 are obtained from the model design information DB 500 of the storage device 203.

[0171] Next, in S152, a process is performed to display on the monitor 300 the score transition display screen 1100 illustrated in FIG. 19, which includes a score transition graph created using the scores for each revision obtained by the process of S151.

[0172] Here, the score transition display screen 1100 shown in Fig. 19 will be described. The score transition display screen 1100 is a screen that displays the transition of the score in chronological order by displaying the above-mentioned score, which is information indicating the reliability of the trained model for each revision, in association with each revision. By referring to the score transition display screen 1100, a model developer who performs the work of updating a trained model can easily understand the transition of the model's reliability brought about by past updates to the trained model.

[0173] The score transition display screen 1100 has a score transition graph display area 1110. The score transition graph display area 1110 is an area for displaying a score transition graph. The score transition graph is a graph that shows the transition of scores for a trained model. This graph is created by smoothly connecting points that correspond to each revision, which is version information for the trained model, and the score of the trained model for each revision, in the order in which the trained model for each revision was created.

[0174] In the example of Figure 19, the score transition graph shown in the score transition graph display area 1110 plots points indicating the scores of the trained model for each revision, and these points are smoothly connected in the order of revisions "1.0," "1.1," "1.2," ...

[0175] While this score transition display screen 1100 is displayed, when the input device 400 is operated so that the mouse pointer 720 points to a point indicating the score for each revision in the score transition graph in the score transition graph display area 1110 or its vicinity, the design information display area 910 appears. The design information display area 910 is an area that displays information about the trained model of the revision whose score is indicated at or near the position pointed to by the mouse pointer 720. The example in FIG. 19 shows a state in which the mouse pointer 720 points to the legend display for revision "1.2," causing design information about the trained model of revision "1.2" to be displayed in the design information display area 910.

[0176] 19, in the process of S152, a score transition graph is created from the score values ​​acquired in the process of S151. The processor 201 causes the monitor 300 to display a score transition display screen 1100 in which the created score transition graph is displayed in a score transition graph display area 1110.

[0177] Next, in S153, a process is performed to acquire an instruction operation for the input device 400. Then, in S154, a process is performed to determine whether the acquired instruction operation is an operation to select a revision of a trained model by selecting a score in the score transition graph in the score transition graph display area 1110.

[0178] If it is determined in the determination process of S154 that the instruction operation is an operation to select a revision, the process proceeds to S155. Then, in S155, the design information display area 910 is popped up, and design information of the trained model of the selected revision is acquired from the model design information DB 500 and displayed in the design information display area 910. Thereafter, the process returns to S153, and the process of acquiring the instruction operation is performed again.

[0179] On the other hand, if it is determined in the determination process of S154 that the instruction operation was not an operation to select a revision, the process proceeds to S156. Then, in S156, a process is performed to determine whether the acquired instruction operation was an operation to deselect a revision by moving the mouse pointer 720 from the score transition graph display area 1110.

[0180] If it is determined in the determination process of S156 that the instruction operation was an operation to cancel the selection of a revision, the process proceeds to S157. Then, in S157, the pop-up display of the design information display area 910 is erased from the score transition display screen 1100, and the original screen is restored. Thereafter, the process returns to S153, and the process of acquiring the instruction operation is performed again.

[0181] On the other hand, if it is determined in the determination process of S156 that the instruction operation was not an operation to cancel the selection of a revision, the process proceeds to S158. Then, in S158, a process is performed to determine whether the acquired instruction operation was a click operation on the "Back" button 860 of the score transition display screen 1100. If it is determined in this determination process that the instruction operation was a click operation on the "Back" button 860, the score transition display screen process is terminated, and the process returns to the original process, that is, the model design information screen process or the sort process.

[0182] On the other hand, if it is determined in the determination process of S158 that the instruction operation is not a click operation on the "back" button 860, the process returns to S153, and the instruction operation on the input device 400 is acquired again.

[0183] [7: Re-learning process]

[0184] Next, the re-learning process will be described with reference to the flowchart of FIG.

[0185] The re-learning process is executed by the processor 201 as the process of S115 when the instruction operation acquired by the process of S113 in the model design information screen process of Fig. 10 is a click operation on the "Create AI model" button 870. This process is also executed by the processor 201 as the process of S125 when the instruction operation acquired by the process of S123 in the sorting process of Fig. 12 is a click operation on the "Create AI model" button 870.

[0186] A model developer who updates a trained model, as part of the work in S21 of Figure 2, refers to the screens displayed by the various processes described above to understand the design intent and process behind the creation of the old version of the AI ​​model. The model developer then performs the review work in S22 and performs the work from S23 onwards based on the results of that review. The re-learning process is the process for the work from S23 onwards.

[0187] 20 starts, first, in S201, a process of acquiring an instruction operation on the input device 400 is performed. Then, in the subsequent determination processes of S202, S204, S206, and S208, a process of determining the instruction content indicated by the instruction operation is performed.

[0188] When it is determined in the determination process of S202 that the instruction operation indicates an instruction to acquire a video, a learning video acquisition process of S203 is performed. This process is a process for acquiring a learning video, and is a process for the additional acquisition of a learning video, which is the process of S24 in the AI ​​model update process shown in Figure 2.

[0189] If it is determined in the determination process of S204 that the instruction operation indicates an instruction to perform annotation, the annotation process of S205 is performed. This process is a process of adding annotations to the learning video, and is a process for adding annotations to the learning video or changing annotations that have already been added, which is the process of S26 in the AI ​​model update process shown in Figure 2.

[0190] If it is determined in the determination process of S206 that the instruction operation indicates an instruction to set learning conditions, the process proceeds to a learning condition setting process of S207. This process is a process for setting learning conditions in machine learning for creating an AI model, and is a process for resetting the learning conditions, which is the process of S28 in the AI ​​model update process shown in Figure 2.

[0191] When the process of S203, S205, or S207 described above is completed, the process returns to S201, and the process of acquiring a new instruction operation is performed again.

[0192] On the other hand, if it is determined in the determination process of S208 that the instruction operation indicates an instruction to execute machine learning, the machine learning process of S209 is performed. This process is a process of performing machine learning for creating an AI model according to the set learning conditions, and is a process for the work of each procedure from S15 to S20 in Figure 1 as re-learning in the update work of the AI ​​model.

[0193] Thereafter, when the machine learning process in S209 is completed, in S210, configuration data for the trained model after re-learning created by the machine learning process is saved in the storage device 203. Then, in the following S211, design information for the trained model after re-learning is stored in the model design information DB 500 of the storage device 203 in association with a revision, which is version information indicating the version of the trained model after re-learning.

[0194] After the process of S211 is completed, this re-learning process is terminated, and the process proceeds to the re-learning support process of FIG. 8, where the process of S101, that is, the process of displaying the model selection screen 700, is performed.

[0195] If the instruction content indicated by the instruction operation cannot be determined by any of the determination processes of S202, S204, S206, and S208, the process returns to S201, and the process of acquiring a new instruction operation is performed again.

[0196] The above processing is the re-learning processing.

[0197] As described above, the relearning support system 1 presents design information about a trained model for each created version (revision) of the trained model, making it easy to appropriately recognize differences in design information for each version of a trained model created in the past. Because the relearning support system 1 is configured in this way, it can support the work of relearning a trained model, and model developers who perform the work of updating a trained model can easily perform this relearning work.

[0198] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications and alternatives to the above-described embodiments may be included. In other words, the components of the above-described embodiments may be modified without departing from the spirit and scope of the invention. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in the embodiments. Some components may be deleted from the components shown in the embodiments, or some components may be added to the components shown in the embodiments. Furthermore, the order of the processing steps shown in the embodiments may be reversed as long as no contradictions are present. In other words, the relearning support system of the present invention can be modified and changed in various ways without departing from the scope of the claims.

[0199] For example, in the embodiment described above, the configuration data of the trained model, the model design information DB 500, and the model-related file information DB 600 are stored separately in the storage device 203 of the computer 200a serving as the control device 200. Alternatively, the design information stored in the model design information DB 500 and the information on various file names stored in the model-related file information DB 600 may be embedded in the configuration data of the corresponding version of the trained model and stored separately in the storage device 203.

[0200] In this specification, the expression "based on A" does not mean "based only on A," but rather "based at least on A," and further means "based at least partially on A." In other words, "based on A" may be based on B in addition to A, or may be based on a part of A. [Explanation of symbols]

[0201] 1. Re-learning support system 100 microscopes 101 Objective Lens 106 Eyepiece 112 Imaging device 113 Projector 120 Eyepiece tube 121 Operation section 130 Zoom Handle 140 Aiming Handle 200 control device 200a Computer 201 processor 202 memory 203 Storage device 204 Reading device 205 Removable storage media 206 Communication Interface 207 Input / Output Interface 208 Bus 300 monitors 400 Input Device 401 Mouse 402 Keyboard 403 Foot Switch 404 Barcode reader 500 Model Design Information DB 600 model related file information DB 700 Model Selection Screen 710 Model selection button 720 Mouse Pointer 730 Video list screen 800 Model design information screen 810 1st display area 820 2nd display area 830 Video Selection Area 840 video display area 851 Sort button 852 Tag information button 853 Performance Comparison Button 854 Score Change Button 860 "Back" button 870 "Create AI model" button 900 Video tag information screen 910 Design information display area 920 Tag information list display area 930 Tag information selection area 1000 model performance comparison screen 1010 Performance comparison display area 1100 Score history display screen 1110 Score transition graph display area

Claims

1. one or more trained models trained using at least one microscopic image; One or more pieces of version information corresponding to each of the trained models, the version of the trained model being indicated; one or more pieces of design information about the trained model, the design information being associated with the one or more pieces of version information; a storage unit that stores the Accepting a selection of at least one of the trained models; Acquire the one or more pieces of version information and the one or more pieces of design information for the selected trained model from the storage unit; Displaying information about the selected trained model in a first display area of ​​a display; The acquired version information and the acquired design information are displayed in association with each other in a second display area of ​​the display. A control unit; A re-learning support system comprising:

2. 2. The re-learning support system according to claim 1, wherein the design information includes at least one of time information, person information, learning information, and character information.

3. the learning information includes the number of inferences and a score; The control unit displays the number of inferences and the score in association with the version information in the second display area.

3. The re-learning support system according to claim 2.

4. The re-learning support system according to claim 2, wherein the design information further includes microscope observation information when the microscope image used to generate the trained model was acquired.

5. The re-learning support system according to claim 1, characterized in that when the trained model having the version information is re-trained, the design information for the trained model after the re-training is associated with the version information indicating the version of the trained model after the re-training and stored in the memory unit.

6. The re-learning support system according to claim 1, wherein the control unit accepts the selection of at least one of the trained models using a selection screen displayed on the display.

7. the at least one microscopic image is an image from one or more image sequences; The storage unit further stores the moving image, The control unit displays the moving image in a third display area of ​​the display.

2. The re-learning support system according to claim 1.

8. the design information includes tag information, The storage unit further stores the tag information in association with the video; The control unit further Accepting a selection of the tag information; The selected tag information is associated with the associated video image and displayed on the display.

8. The re-learning support system according to claim 7.

9. The re-learning support system according to claim 1, characterized in that the control unit changes the display order of the associated display of the version information and the design information in the second display area based on the design information.

10. The control unit further displays, on the display, a plurality of pieces of version information corresponding to the selected trained model and information indicating performance of the version of the trained model indicated by the version information. The re-learning support system according to claim 1.

11. The control unit further Accepting a selection of the plurality of versions of information; The re-learning support system according to claim 10, wherein the information indicating the performance of each of the selected plurality of version information is displayed on the display in a comparative display format.

12. The re-learning support system according to claim 1, characterized in that the control unit further causes the display to display, in chronological order, a plurality of pieces of version information corresponding to the selected trained model and a score indicating the reliability of the trained model of the version indicated by the version information.

13. Accepting a selection of one or more trained models trained using at least one microscopy image; acquires, from a storage unit that stores the one or more trained models, one or more pieces of version information indicating versions of the trained models that are associated with each of the trained models, and one or more pieces of design information about the trained models that are associated with the one or more pieces of version information; Displaying information about the selected trained model in a first display area of ​​a display; The acquired version information and the acquired design information are displayed in association with each other in a second display area of ​​the display. A re-learning support method characterized in that the above steps are carried out by a computer.

14. Accepting a selection of one or more trained models trained using at least one microscopy image; acquires, from a storage unit that stores the one or more trained models, one or more pieces of version information indicating versions of the trained models that are associated with each of the trained models, and one or more pieces of design information about the trained models that are associated with the one or more pieces of version information; Displaying information about the selected trained model in a first display area of ​​a display; The acquired version information and the acquired design information are displayed in association with each other in a second display area of ​​the display. A re-learning support program characterized by causing a computer to perform processing.

Images