Display method, program and display device

Abstract

Provided is a method and apparatus that allows a user to develop a machine learning model without necessarily needing specialized knowledge of AI and skills such as programming. [Solution] The display method disclosed herein includes a step in which a computer displays on a display device an operation screen that displays a machine learning procedure for image segmentation, with hyperparameters for controlling the learning process fixed in advance and the hyperparameters unable to be changed from the screen, and that has a plurality of buttons for accepting user instructions to cause the computer to execute processing in accordance with the procedure.

Description

[Technical field]

[0001] The present invention relates to a display method, a program, and a display device. [Background technology]

[0002] With the evolution of artificial intelligence (AI), various applications of AI have been proposed. For example, JP 2021-72490 A (Patent Document 1) discloses an image processing device that includes a learning means that executes machine learning processing using information contained in input image data, and an output means that outputs an estimation result based on information contained in the image data using a trained model created by the machine learning processing. The image processing device is characterized in that when the input image data contains confidential information, the confidential information is not learned by machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-72490 Summary of the Invention [Problem to be solved by the invention]

[0004] It is generally understood that in order to develop a machine learning model such as that disclosed in the above-mentioned literature, programming skills are required in addition to knowledge of machine learning, including deep learning. However, if it becomes possible to create machine learning models without necessarily requiring such specialized knowledge and skills, more people can be provided with the opportunity to create machine learning models. Therefore, it is expected that various advantages will arise, such as further advances in the use of AI.

[0005] An object of the present disclosure is to provide a method and apparatus that allows a user to develop a machine learning model without necessarily requiring specialized knowledge of AI and skills such as programming. [Means for solving the problem]

[0006] The display method according to the present disclosure includes a step of causing a computer to display on a display device a screen that displays a machine learning procedure for image segmentation, with hyperparameters for controlling the learning process fixed in advance and the hyperparameters unable to be changed from the screen, and that has a plurality of buttons arranged thereon for accepting user instructions for causing the computer to execute processing in accordance with the procedure.

[0007] The program of the present disclosure causes a computer to execute a step of displaying on a display device a screen that displays a machine learning procedure for image segmentation, with hyperparameters for controlling the learning process fixed in advance and the hyperparameters unable to be changed from the screen, and that has a plurality of buttons that accept user instructions for causing the computer to execute processing in accordance with the procedure.

[0008] The display device according to the present disclosure comprises a display unit, a display control unit configured to control the display unit, and a machine learning unit configured to perform machine learning using a predetermined neural network, and the display control unit is configured to cause the display unit to display a screen on which a machine learning procedure for image segmentation is displayed, with hyperparameters for controlling the learning process being fixed in advance and the hyperparameters being unable to be changed from the screen, and on which a plurality of buttons are arranged to accept user instructions for causing the machine learning unit to perform processing in accordance with the procedure. Effect of the Invention

[0009] According to the present disclosure, it is possible to provide a user with an environment in which they can develop machine learning models without necessarily needing specialized knowledge of AI and skills such as programming. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram illustrating a schematic functional block of a display device according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing an overview of a machine learning procedure according to an embodiment of the present disclosure. [Diagram 3] FIG. 3 is a diagram showing an example of a display screen displayed by the display device according to the present embodiment. [Figure 4] FIG. 4 is a flowchart showing the flow of display processing by the display device according to this embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of the training data. [Figure 6] FIG. 6 is an example of a display indicating that the machine learning unit is currently learning. [Figure 7] FIG. 7 is an example of a log screen showing that learning by the machine learning unit has been completed. [Figure 8] FIG. 8 is a diagram showing an example of images contained in a folder in which image data for inference is stored. [Figure 9] FIG. 9 is an example of a display indicating that the machine learning unit is performing inference. [Figure 10] FIG. 10 is a diagram showing an example of an inference result by the machine learning unit and a correction of the inference result. [Figure 11] FIG. 11 is an example of a display indicating that the corrected inference result will be added as training data. [Figure 12] FIG. 12 is an example of a display indicating that the machine learning unit is undergoing relearning. [Figure 13] FIG. 13 is a diagram showing an example of an original image used for inference by the machine learning model according to the present embodiment. [Figure 14] FIG. 14 is a diagram showing the result of inference by a machine learning model developed using the display device according to the present embodiment. [Figure 15] FIG. 15 is a schematic diagram of an example of a support display. [Figure 16] FIG. 16 is a schematic diagram of another example of the support display. [Figure 17] FIG. 17 is a schematic diagram of yet another example of the support display. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [Description of the embodiments of the present disclosure] First, the embodiments of the present disclosure will be listed and described.

[0012] (1) A display method according to an embodiment of the present disclosure includes a step of causing a computer to display on a display device a screen that displays a machine learning procedure for image segmentation, with hyperparameters for controlling a learning process being fixed in advance and the hyperparameters not being changeable from the screen, and that has a plurality of buttons arranged thereon for receiving user instructions to cause the computer to execute processing in accordance with the procedure.

[0013] According to the above configuration, the user can operate multiple buttons on the screen in a procedure while the hyperparameters for controlling the learning process cannot be changed, and the computer can perform machine learning without making complex adjustments related to the creation of a machine learning model. Specifically, a user without expert knowledge can proceed with model creation without being aware of the existence of hyperparameters specified in advance by a user with expert knowledge of machine learning. It is generally understood that adjustment of hyperparameters is necessary in the creation of a machine learning model. However, the adjustment of hyperparameters requires knowledge of hyperparameter tuning algorithms and programming skills, for example. By fixing the hyperparameters in advance, a certain degree of accuracy can be ensured for learning by the machine learning model, while simplifying the user's operation. Therefore, even a user without expert knowledge of AI and skills such as programming can create a machine learning model.

[0014] (2) In the display method of (1) above, the multiple buttons include a first button for specifying to the computer the location of the data selected by the user, a second button for accepting a user instruction to instruct the computer to train a machine learning model using the teacher data specified by operation of the first button, a third button for accepting a user instruction to instruct the computer to perform inference using the inference data with the trained machine learning model, and a fourth button for accepting a user instruction to instruct the computer to add data based on the results of the inference as teacher data.

[0015] According to the above configuration, a user can cause a computer to execute machine learning (learning and inference) by operating the first to fourth buttons in sequence according to the procedure.

[0016] (3) In the display method of (2) above, among the first button to the fourth button, at least the second button, the third button, and the fourth button are arranged in a line in that order.

[0017] According to the above configuration, information regarding the machine learning procedure can be provided to the user so that the user can intuitively understand the machine learning procedure.

[0018] (4) The display method according to (2) or (3) above further comprises the step of displaying that the computer is currently executing a process in response to the operation of each of the first to fourth buttons.

[0019] According to the above configuration, information that each step of the machine learning process is being executed is presented to the user, thereby enabling the user to grasp how far the machine learning procedure has progressed.

[0020] (5) The display method according to any one of (1) to (4) above, further comprising the step of: displaying information for informing the user of a button to be operated next.

[0021] According to the above configuration, it is possible to guide a user so that the user has the computer perform machine learning in accordance with the correct procedure.

[0022] (6) The program of the present disclosure causes a computer to execute a step of displaying on a display device a screen that displays a machine learning procedure for image segmentation, with hyperparameters for controlling the learning process fixed in advance and the hyperparameters unable to be changed from the screen, and that has a plurality of buttons for accepting user instructions to cause the computer to execute processing in accordance with the procedure.

[0023] According to the above configuration, the user can operate multiple buttons on the screen in a sequential order, which allows the computer to execute machine learning. Even users who do not have specialized knowledge of AI or programming skills can create a machine learning model.

[0024] (7) A display device according to the present disclosure includes a display unit, a display control unit configured to control the display unit, and a machine learning unit configured to perform machine learning of a predetermined neural network, wherein the display control unit is configured to cause the display unit to display a screen on which a machine learning procedure for image segmentation is displayed, with hyperparameters for controlling the learning process being fixed in advance and the hyperparameters being unable to be changed from the screen, and on which a plurality of buttons are arranged to accept user instructions for causing the machine learning unit to perform processing in accordance with the procedure.

[0025] According to the above configuration, the user can operate multiple buttons on the screen in a sequential order, which allows the computer to execute machine learning. Even users who do not have specialized knowledge of AI or programming skills can create a machine learning model.

[0026] [Details of the embodiment of the present disclosure] <Device configuration> 1 is a block diagram showing a schematic functional block of a display device 1 according to an embodiment of the present disclosure. The display device 1 is mainly realized by a computer.

[0027] As shown in FIG. 1, the display device 1 includes a display processing unit 10 and an image processing unit 20. The display processing unit 10 includes a display unit 11 and a display control unit 12. The display unit 11 is typically a display and displays various information. In particular, in the embodiment of the present disclosure, the display unit 11 displays a screen for a user to operate the display device 1. On the screen, a plurality of buttons are arranged that display a procedure of machine learning by a computer and accept a user's instruction to cause the computer to execute a process according to the procedure. The display unit 11 may be a touch panel. The touch panel can have a function of an input unit in addition to a function as a display unit.

[0028] The display control unit 12 is a control circuit for controlling the operation of the display unit 11 so that the display unit 11 displays information. The display control unit 12 can be realized by a semiconductor integrated circuit such as an LSI (Large Scale Integration).

[0029] The image processing unit 20 includes an input unit 21, a storage unit 22, a communication unit 23, and a machine learning unit 24.

[0030] The input unit 21 receives instructions from a user. The input unit 21 may include input devices such as a mouse, a keyboard, a pen tablet, or a pen-type input device used in a device equipped with a touch panel. Therefore, as described above, a touch panel may be used as the display unit 11 and the input unit 21.

[0031] The storage unit 22 stores various information such as a machine learning model, teacher data, the contents of calculations by the machine learning unit 24, and programs. The storage unit 22 includes a main storage device such as a random access memory (RAM) and a read only memory (ROM), and a secondary storage device such as a hard disk drive (HDD) or a solid state memory. The information stored in the secondary storage device can be acquired from outside the display device 1 via at least one of the input unit 21 and the communication unit 23.

[0032] The communication unit 23 is an interface circuit that is responsible for communication between the display device 1 and an external device (not shown in FIG. 1). The communication by the display device 1 may be either wired communication or wireless communication, or both.

[0033] The machine learning unit 24 is configured to execute machine learning using a predetermined neural network. In this embodiment, the machine learning unit 24 can be realized by an arithmetic circuit such as a central processing unit (CPU) or a graphics processing unit (GPU).

[0034] <Machine learning> Image analysis can be cited as one application of image processing by the image processing unit 20. For example, segmentation is used in image analysis. Segmentation is a technique in machine learning that divides an image into multiple objects.

[0035] Semantic segmentation is an example of segmentation performed by the machine learning unit 24. Semantic segmentation refers to a deep learning algorithm that classifies all pixels in an image and associates a label or category with each pixel. In semantic segmentation, a label is assigned to each pixel in an image. This makes it possible to detect amorphous regions with high accuracy.

[0036] Examples of semantic segmentation include FCN (Fully Convolutional Network), U-Net, SegNet, etc. In the embodiment described below, segmentation by U-Net is applied as an example.

[0037] U-Net is a type of FCN that includes an encoder that performs downsampling and a decoder that performs upsampling. U-Net is a neural network that consists only of convolutional layers and pooling layers. U-Net has the characteristic of exhibiting high segmentation performance even with a small amount of training data.

[0038] To perform learning and inference of U-Net, good quality training data is required. In this embodiment, active learning is introduced into the annotation process to efficiently prepare training data. The combination of U-Net and active learning is excellent in terms of, for example, the wide range of applications.

[0039] Fig. 2 is a diagram showing an overview of a machine learning procedure according to an embodiment of the present disclosure. Fig. 2 shows an active learning procedure according to an embodiment of the present disclosure. First, in step S1, a small amount (for example, one pair) of image data and an annotation image of the image data are prepared as teacher data.

[0040] In step S2, the machine learning unit 24 trains the U-Net using the training data. In the first training, the machine learning unit 24 trains the U-Net using the training data prepared in step S1.

[0041] Next, in step S3, the machine learning unit 24 performs inference using data other than the teacher data, thereby generating image data as a result of the inference.

[0042] In step S4, the image data corresponding to the inference result is corrected. Specifically, the image data corresponding to the inference result of the machine learning unit 24 is confirmed by the user. If necessary, the user corrects the image data.

[0043] In step S5, image data corresponding to the inference result (if the inference result is corrected by another process after U-Net inference, the corrected inference result) is added as new training data. From step S5, the flow returns to step S2, and the machine learning unit 24 re-trains the U-Net using the added training data.

[0044] 2, in this embodiment, a cycle of learning, inference, correction of the inference result, and learning is repeated a required number of times. That is, the machine learning model is repeatedly given small amounts of training data to learn.

[0045] In deep learning techniques such as segmentation using U-Net, it is common to adjust hyperparameters to improve the performance of machine learning models. Hyperparameters are necessary to control the behavior of various algorithms, and typical examples include the number of epochs, learning rate, mini-batch size, number of layers, number of neurons per layer, etc. Adjusting hyperparameters can improve the performance of machine learning models and suppress overlearning.

[0046] Tuning hyperparameters requires, for example, knowledge of hyperparameter tuning algorithms, programming skills, etc. Therefore, there is a problem that ordinary people who do not have such knowledge and skills cannot be involved in creating machine learning models.

[0047] On the other hand, in this embodiment, the hyperparameters of the machine learning model are fixed in advance, and the hyperparameters cannot be changed from the screen. By fixing the hyperparameters of the machine learning model, it is possible to eliminate the need for complex adjustments of the machine learning model. This makes it possible to reduce the difficulty of creating a machine learning model.

[0048] Specifically, the work required for the user of the display device 1 is to prepare data necessary for the learning and inference of the display device 1 (machine learning unit 24), and to input to the display device 1 an instruction for causing the machine learning unit 24 to perform learning and inference. In this embodiment, the display device 1 displays to assist the user in these operations. An example of the display by the display device 1 will be described below in accordance with the procedure shown in FIG. 2.

[0049] In addition, when the hyperparameters are fixed, there is a concern that the accuracy of image recognition by the machine learning model may decrease. However, depending on the target of image analysis, it may not be necessary to require high recognition accuracy. As described below, in one embodiment of the present disclosure, since the recognition is of a relatively simple figure in an image, the accuracy of image recognition does not need to be higher than necessary. Therefore, according to this embodiment, even a user who does not have specialized knowledge about AI and skills such as programming can create a machine learning model. In addition, even if the level of the highest accuracy that the model can reach is lowered by fixing the hyperparameters, the industrial advantage of being able to reduce the hyperparameter adjustment process required to achieve it and enabling users who do not have specialized knowledge about machine learning to work is obtained.

[0050] <Display example> FIG. 3 is a diagram showing an example of a display screen displayed by the display device 1 according to the present embodiment. As shown in FIG. 3, the display screen 100 includes buttons 101 to 105. The buttons 101 to 105 are buttons selected by the user, and are buttons that display a procedure of machine learning by the machine learning unit 24 and accept a user's instruction to cause the machine learning unit 24 (computer) to execute processing according to the procedure. Note that in the following, "button operation" refers to an action in which a user selects a corresponding button by operating an input device such as a pointing device. A typical example of "button operation" is moving a mouse pointer to a corresponding button and clicking the mouse button.

[0051] Button 101 is a first button for specifying the location of data selected by the user on the display device 1 (computer). When the user selects a folder containing target data, the path including the folder name of the folder is displayed in the folder display window 106.

[0052] Button 102 is a second button that accepts a user's instruction to instruct machine learning unit 24 to learn a machine learning model using the teacher data specified by operating button 101.

[0053] Button 103 is a third button for accepting a user's instruction to instruct machine learning unit 24 to perform inference using inference data for a trained machine learning model.

[0054] Button 104 is a fourth button for accepting a user's instruction to instruct display device 1 (computer) to add data based on the result of inference as teacher data.

[0055] Button 105 is a button for instructing machine learning unit 24 to end machine learning.

[0056] The user can make the machine learning unit 24 perform machine learning by operating the buttons 101 to 105 on the display screen 100 according to the procedure. Therefore, even a user who does not have specialized knowledge about AI and skills such as programming can create a machine learning model. In particular, the buttons 102 to 105 are arranged in a line from left to right in this order. The user can make the machine learning unit 24 perform machine learning by operating the buttons 102, 103, and 104 in order according to the procedure. According to this embodiment, it is possible to provide a display screen that allows the user to intuitively understand the procedure for making the machine learning unit 24 perform machine learning.

[0057] Furthermore, according to this embodiment, the function assigned to each button is written in hiragana. This makes it easier for users to understand the function of each button, making it possible to provide a display device that is easy to use for more people.

[0058] 3 is just an example. Other button arrangements may be used as long as the buttons are arranged in a way that allows the user to intuitively understand the procedure for causing the machine learning unit 24 to learn and infer.

[0059] 4 is a flowchart showing the flow of display processing by the display device 1 according to the present embodiment. The display device 1 reads out a program stored in the storage unit 22 and executes the program, thereby performing processing according to the flow shown in FIG.

[0060] In step S11, the display control unit 12 causes the display unit 11 to display the display screen 100 shown in FIG. 3. In step S12, the machine learning unit 24 determines whether any of the buttons on the display screen 100 has been selected. Specifically, the machine learning unit 24 determines whether any of the buttons 101 to 105 on the display screen 100 has been selected based on a signal received from the input unit 21. For example, a user moves a pointer of a pointing device (such as a mouse) onto a corresponding button in the display screen 100 to select the button (for example, clicks a mouse button). A signal is input to the input unit 21 by operating the pointing device. The machine learning unit 24 identifies the button selected by the user from the buttons 101 to 105 based on the signal input from the input unit 21.

[0061] If none of buttons 101 to 105 is selected (NO in step S12), the process returns to step S11. If any of buttons 101 to 105 is selected (YES in step S12), in step S13, machine learning unit 24 executes processing according to the selected button, and display control unit 12 controls display unit 11 to display a screen according to the processing of machine learning unit 24. When the processing of step S13 ends, the process returns to step S11.

[0062] Next, an example of machine learning according to this embodiment will be described along with the transition of the display screen by the user's operation.

[0063] <Example of machine learning> (1) Selection of training data Fig. 5 is a diagram showing an example of teacher data. As shown in Fig. 5, an original image 31 and an annotation image 32 obtained by annotating the original image 31 are stored in a teacher data folder 30 formed in the storage unit 22.

[0064] 5 is a CT image of the solder that connects the circuit board and the terminals. A region 33 in the original image 31 is a region where a void in the solder is imaged.

[0065] Annotated image 32 is an image obtained after annotation has been applied to voids in region 33 of original image 31. Note that various known techniques can be used for annotation, and therefore detailed description will not be repeated here.

[0066] (2) Learning The user operates button 101 shown in Fig. 3. As a result, a folder selection screen (not shown in Fig. 3) on which, for example, a folder name is displayed is displayed on display unit 11. The user selects teacher data folder 30 from the folder selection screen. As a result, the path of teacher data folder 30 is displayed in folder display window 106.

[0067] Next, the user operates button 102. As a result, a user instruction for instructing the machine learning unit 24 to perform learning is input to the machine learning unit 24 via the input unit 21. In response to the instruction, the machine learning unit 24 reads out the teacher data from the folder (teacher data folder 30) displayed in the folder display window 106, and uses the teacher data to train the U-Net.

[0068] Fig. 6 is an example of a display indicating that the machine learning unit 24 is learning. As shown in Fig. 6, while the machine learning unit 24 is learning, a window 110 indicating "Studying" is displayed on the screen. Therefore, the user can confirm that the machine learning unit 24 is learning using the teacher data.

[0069] Similar to the notation of the buttons, the message is written in hiragana in the window 110. In this way, the message is written in hiragana in the window that displays that the processing of the machine learning unit 24 is being executed. This makes it easier for the user to understand the meaning of the message.

[0070] When the learning by the machine learning unit 24 is completed, the display control unit 12 may control the display of the screen by the display unit 11 so that the window 110 is automatically closed. This allows the user to confirm that the learning by the machine learning unit 24 is completed.

[0071] The display control unit 12 may control the display of the screen by the display unit 11 so as to display a log screen showing a log indicating the progress of the processing by the machine learning unit 24. By updating the log, the user can check the progress of the learning by the machine learning unit 24.

[0072] 7 is an example of a log screen indicating that learning by the machine learning unit 24 has finished. A message of completion ("finish training" in the example of FIG. 7) is displayed on the log screen 121, allowing the user to confirm that learning of the U-Net by the machine learning unit 24 has finished. The display control unit 12 may control the display of the screen by the display unit 11 so that the log screen 121 is automatically closed when learning by the machine learning unit 24 has finished.

[0073] (3) Reasoning 3, a folder selection screen (not shown in FIG. 3) displaying, for example, a folder name is displayed on the display unit 11. The user selects a folder in which image data to be used for inference by the machine learning unit 24 is stored. As a result, the path of the folder in which the image data for inference is stored is displayed in the folder display window 106.

[0074] Fig. 8 is a diagram showing an example of images contained in a folder in which image data for inference is stored. In the example shown in Fig. 8, 10 pieces of image data are stored in folder 35. These pieces of image data are different data from the teacher data (original image 31) used for learning by machine learning unit 24 in step "(2) Learning" above.

[0075] Next, the user operates button 103 shown in Fig. 3. As a result, a user instruction for instructing the machine learning unit 24 to perform inference is input to the machine learning unit 24 via the input unit 21. In response to the instruction, the machine learning unit 24 reads image data from the folder (folder 35) displayed in the folder display window 106, and performs inference using the image data.

[0076] Fig. 9 is an example of a display indicating that the machine learning unit 24 is performing inference. As shown in Fig. 9, while the machine learning unit 24 is performing inference, a window 111 displaying "watering" is displayed on the screen. This allows the user to confirm that the machine learning unit 24 is performing inference. As with window 110, a message is displayed in hiragana in window 111.

[0077] As in the example shown in FIG. 7, the display control unit 12 may control the display of the screen by the display unit 11 so that a log screen 122 that displays a log of processing by the machine learning unit 24 is displayed.

[0078] When the inference by the machine learning unit 24 ends, the display control unit 12 may control the display of the screen by the display unit 11 so that the window 111 is automatically closed. When the log screen 122 is displayed during the inference by the machine learning unit 24, the display control unit 12 may control the display of the screen by the display unit 11 so that the log screen 122 is automatically closed when the inference by the machine learning unit 24 ends.

[0079] (4) Correction of inference results FIG. 10 is a diagram showing an example of an inference result by the machine learning unit 24 and a correction of the inference result. An original image 41 shown in FIG. 10 corresponds to one of the ten images shown in FIG. 8. An area 41A in the original image 41 corresponds to an area where an image of a void in the solder is captured. When the area 41A is enlarged, it can be confirmed that an image of a void 41B and a void 41C is captured within the area 41A. The void 41C is smaller than the void 41B. Furthermore, the image of the void 41C is unclear compared to the image of the void 41B.

[0080] Image 42 is image data output as an inference result by machine learning unit 24. Region 42A in image 42 corresponds to region 41A of original image 41. Region 42A includes region 42B made up of white pixels. Region 42B corresponds to void 41B. On the other hand, the inference result does not include a region corresponding to void 41C. That is, in the inference result by the machine learning unit, void 41C is not recognized.

[0081] 10, a person modifies image 42 so that the area corresponding to void 41C is included in image 42. For example, a general paint tool can be used for the modification. Note that the person modifying image 42 does not need to be the same as the user who operates display device 1.

[0082] Image 43 is an image after image 42 has been corrected, and corresponds to the result of correcting the inference result of machine learning unit 24. Regions 43B and 43C in region 43A correspond to voids 41B and 41C, respectively, in original image 41. That is, by correcting image 42, white region 43C corresponding to void 41C is added to image 42.

[0083] (5) Adding training data When the user operates the button 101 shown in Fig. 3, for example, a folder selection screen (not shown in Fig. 3) displaying a folder name is displayed on the display unit 11. The user selects a folder in which to store the corrected inference result. As a result, the path of the folder in which to store the corrected inference result is displayed in the folder display window 106.

[0084] FIG. 11 is an example of a display indicating that the corrected inference result will be added as training data. When the user operates button 104, image processing unit 20 adds training data to memory unit 22 in accordance with the user's instructions. As shown in FIG. 11, while the training data is being added, window 112 displaying "Adding" is displayed on the screen. This allows the user to confirm that the training data has been added. As with windows 110 and 111, window 112 displays a message written in hiragana.

[0085] (6) Relearning The user operates button 102. As a result, a user instruction for instructing the machine learning unit 24 to perform learning is input to the machine learning unit 24 via the input unit 21. In response to the instruction, the machine learning unit 24 performs learning of the U-Net using the teacher data.

[0086] FIG. 12 is an example of a display indicating that the machine learning unit 24 is re-learning. As shown in FIG. 12, while the machine learning unit 24 is learning, a window 113 with the words "studying" is displayed on the screen. The screen display shown in FIG. 12 is basically the same as the screen display exemplified in FIG. 6, so detailed description will not be repeated. Note that a log screen (not shown) for displaying a log indicating the progress of processing by the machine learning unit 24 may be displayed on the display unit 11. In this case, the display control unit 12 may control the display of the log screen so that the log screen is automatically closed when the learning of the machine learning unit 24 is completed.

[0087] After the learning is completed, the image data for inference by the machine learning unit 24 can be selected again according to the procedure of "(3) Inference" to cause the machine learning unit 24 to perform inference. By repeating the learning and inference by the machine learning unit 24 in the above manner, a machine learning model can be developed.

[0088] <Inference result> Fig. 13 is a diagram showing an example of an original image used for inference by the machine learning model according to the present embodiment. Original image 44 shown in Fig. 13 is a CT image of solder connecting a circuit board and a terminal, similar to original image 31 shown in Fig. 5. Original image 44 is an image that is not included in either the teacher data at the time of initial learning or the additional teacher data.

[0089] Region 44A in original image 44 is a region where a void in the solder is captured. When region 44A is enlarged, it can be seen that images of multiple voids are included within region 44A. The following description focuses on an image capturing void 44B among the images of the multiple voids.

[0090] FIG. 14 is a diagram showing the result of inference by a machine learning model developed using the display device according to the present embodiment. In FIG. 14, an image 45 is an image corresponding to the first inference result. An image 46 is an image corresponding to the inference result (second inference result) by the machine learning model after re-learning. An area 45A in the image 45 and an area 46A in the image 46 correspond to an area 44A (see FIG. 13) in the original image 44. An area 45B and an area 46B correspond to the result of recognizing the image of a void 44B contained in the original image 44 by the trained machine learning model. As can be seen from the comparison between the shapes of the areas 45B and 46B, the second inference result has improved the irregularity of the shape of the image recognized by the machine learning model compared to the first inference result. This means that the accuracy of the inference has improved.

[0091] <Example of support display> The screen shown in FIG. 3 is an example of a display screen according to this embodiment. The display screen may include a display for supporting a user's operation. The user can make the machine learning unit 24 execute machine learning by operating a button according to the support display. The user does not need to remember the procedure for making the machine learning unit 24 execute machine learning. Below, an example of a display for supporting a user's operation is shown.

[0092] Fig. 15 is a schematic diagram of an example of a support display. In the example shown in Fig. 15, when one process (for example, learning) by the image processing unit 20 is completed, the button that the user should operate next (in the example of Fig. 15, the "water bottle" button 103) is displayed in a color different from the other buttons. The display control unit 12 may control the display of the buttons 101 to 105 by the display unit 11 so that the buttons that the user should operate blink in order. The display control unit 12 may control the display by the display unit 11 so that the display color of the button that the user should operate next is changed and the button blinks.

[0093] Fig. 16 is a schematic diagram of another example of the support display. In the example shown in Fig. 16, a pointer 131 is displayed to guide the user to the button to be operated next (the "water bottle" button 103 in the example of Fig. 16). In the example of Fig. 16, the shape of the pointer 131 is an arrow. However, the shape of the pointer 131 is not limited to this. However, it is preferable that the pointer 131 is an icon that is easily distinguishable from a pointer indicating the operation of a pointing device such as a mouse.

[0094] Fig. 17 is a schematic diagram of yet another example of the support display. In the example shown in Fig. 17, a window 132 is displayed to guide the user on the next operation. For example, when the user should next operate the "water bottle" button 103, a window 132 is displayed with a message saying "Please select water bottle."

[0095] The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims rather than the above-described embodiments, and it is intended to include the equivalent meaning of the claims and all modifications within the scope of the claims. [Explanation of symbols]

[0096] 1 Display device 10 Display processing section 11 Display section 12 Display control section 20 Image processing section 21 Input section 22 Memory section 23 Communications Department 24 Machine Learning Department 30 Teacher data folder 31,41,44 Original image 32 Annotation images 33,41A,42A,42B,43A,43B,43C,44A,45A,45B,46A,46B area 35 Folders 41B, 41C, 44B Void 42,43,45,46 Images 100 display screens 101~105 Buttons 106 Folder View Window 110~113,132 windows 121,122 Log screen 131 Pointer S1~S5, S11~S13 steps

Claims

1. A display method comprising a step of causing a computer to display on a display device a screen displaying a machine learning procedure for semantic segmentation of an image, with hyperparameters for controlling a learning process being fixed in advance and the hyperparameters being unable to be changed from the screen, and with a plurality of buttons arranged for accepting user instructions for causing the computer to execute processing in accordance with the procedure, wherein the processing in accordance with the machine learning procedure includes learning a machine learning model using teacher data, inference using the machine learning model, and adding new teacher data based on the results of the inference.

2. The plurality of buttons include: a first button for designating data selected by the user to the computer; a second button for receiving an instruction from the user to instruct the computer to learn a machine learning model using teacher data specified by operating the first button; and a third button for receiving an instruction from the user to instruct the computer to perform inference using inference data for the trained machine learning model; and a fourth button for receiving an instruction from the user to instruct the computer to add data based on a result of the inference as training data. The display method according to claim 1 .

3. The display method according to claim 2 , wherein at least the second button, the third button, and the fourth button among the first button to the fourth button are arranged in a line in this order.

4. 3. The display method according to claim 2, further comprising the step of displaying that a process of the computer corresponding to each of the operations of the first button to the fourth button is being executed.

5. The display method according to claim 1 , further comprising the step of: displaying a message to guide the user to a button to be operated next.

6. On the computer, A program that executes a step of displaying on a display device a screen that displays a machine learning procedure for semantic segmentation of an image, and has a plurality of buttons arranged thereon for accepting user instructions to cause the computer to execute processing in accordance with the procedure, with hyperparameters for controlling a learning process being fixed in advance and the hyperparameters being unable to be changed from the screen, wherein the processing in accordance with the machine learning procedure includes learning a machine learning model using teacher data, inference using the machine learning model, and adding new teacher data based on the results of the inference.

7. A display unit; A display control unit configured to control the display unit; A machine learning unit configured to perform machine learning using a predetermined neural network, The display control unit is A display device configured to display on the display unit a screen that displays the machine learning procedure for semantic segmentation of an image, and has a plurality of buttons that accept user instructions for causing the machine learning unit to execute processing in accordance with the procedure, while hyperparameters for controlling the learning process are fixed in advance and the hyperparameters cannot be changed from the screen, and the processing in accordance with the machine learning procedure includes learning a machine learning model using teacher data, inference using the machine learning model, and adding new teacher data based on the results of the inference.

Images