Image processing device, image processing method, and program

Abstract

The image processing apparatus includes: an image acquisition unit configured to acquire a target image indicating a target position specified by a user; a contour line prediction unit configured to predict a contour line in the vicinity of the target position on the basis of a trained model that learns the relationship between the position in the image and the contour line in the vicinity of the position; and an image output unit configured to output a teacher image indicating a prediction result of the contour line.

Description

Technical Field

[0001] The present invention relates to an image processing device, an image processing method and a program. Background Art

[0002] A technique for image analysis using a machine learning model is known. In order to analyze images with high accuracy using a machine learning model, a large number of high-accuracy teacher images are required.

[0003] For example, Patent Document 1 discloses a teacher data production assisting method for producing teacher data used when constructing a training model for image processing using machine learning.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2020-126414 Summary of the invention

[0005] <Problems to be Solved by the Invention>

[0006] However, in the past, many manual operations were required to create a teacher image, which required a lot of manpower and time. Although there are annotation tools that automatically create teacher images for input images, they are not suitable for images with complex contour shapes.

[0007] One aspect of the present invention is to efficiently produce teacher images for machine learning.

[0008] <Methods used to solve the problem>

[0009] The present invention includes the following configurations.

[0010] [1] An image processing device comprising:

[0011] an image acquisition unit configured to acquire a target image showing a target position designated by a user;

[0012] a contour line prediction unit configured to predict a contour line near the target position based on a trained model that has learned a relationship between a position in an image and a contour line near the position; and

[0013] The image output unit is configured to output a teaching image for showing a prediction result of the contour line.

[0014] [2] The image processing device described in [1] above, wherein:

[0015] The image processing device repeatedly performs:

[0016] acquiring, by the image acquisition unit, the target image showing the new target position; and

[0017] The contour line prediction unit predicts the contour line near the new target position.

[0018] [3] The image processing device described in [1] or [2] above, wherein:

[0019] The trained model is a model that learns the relationship between the area and the contour line near the area based on training data including a training image that draws an area showing a specified brightness and a correct image that shows the contour line near the area contained in the training image.

[0020] [4] The image processing device described in [3] above, wherein:

[0021] The region is drawn at a position randomly selected from the contour lines contained in the training image.

[0022] [5] The image processing device described in [3] or [4] above, wherein:

[0023] The training data includes: the training image obtained by extracting a portion of a basic image including a contour line; and a correct answer image obtained by extracting a portion of the contour line image showing the contour line included in the basic image.

[0024] [6] An image processing method, wherein the following steps are performed by a computer:

[0025] acquiring a target image showing a target location specified by a user;

[0026] Predicting a contour line near the target location based on a trained model that learns a relationship between a location in an image and a contour line near the location; and

[0027] A teacher image showing a prediction result of the contour line is output.

[0028] [7] A program for causing a computer to execute the following steps:

[0029] acquiring a target image showing a target location specified by a user;

[0030] Predicting a contour line near the target location based on a trained model that learns a relationship between a location in an image and a contour line near the location; and

[0031] A teacher image showing a prediction result of the contour line is output.

[0032] <Effects of the Invention>

[0033] According to one aspect of the present invention, teacher images for machine learning can be efficiently produced. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A diagram showing an example of a target image.

[0035] Figure 2 This is a diagram showing an example of a teacher image.

[0036] Figure 3 This is a diagram showing an example of a contour line image.

[0037] Figure 4 This is a block diagram showing an example of the overall structure of the teacher image production support system.

[0038] Figure 5 This is a block diagram showing an example of the hardware configuration of a computer.

[0039] Figure 6 This is a block diagram showing an example of the functional structure of the teacher image production support system.

[0040] Figure 7 This is a flowchart showing an example of a teacher image production support method.

[0041] Figure 8 This is a flowchart showing an example of training data generation processing.

[0042] Fig. 9 A diagram showing an example of a training image.

[0043] Fig.10 This is a diagram showing an example of a correct answer image.

[0044] Fig.11 FIG. 1 is a diagram showing an example of a training image showing a training position.

[0045] Fig.12 FIG. 1 is a diagram showing an example of a correct answer image from which a correct answer contour line is extracted.

[0046] Fig.13 This is a flowchart showing an example of model training processing.

[0047] Fig.14 It is a sequence diagram showing an example of the teacher image generation process.

[0048] Fig.15 It is a diagram showing an example of a verification image.

[0049] Fig.16 This is a diagram showing an example of a contour line image.

[0050] Fig.17This is a diagram showing an example of prediction results. DETAILED DESCRIPTION

[0051] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that in this specification and the drawings, components having substantially the same functional structure are given the same reference numerals, and repeated descriptions are omitted.

[0052] [Implementation Method]

[0053] One embodiment of the present invention is a teacher image production assistance system that assists in the production of teacher images for machine learning. The teacher image in this embodiment is used to train a machine learning model that analyzes an image. The teacher image in this embodiment is an image that shows a correct solution of a contour line in an image that is an analysis object. The teacher image in this embodiment can be used to train a machine learning model that performs tasks such as edge detection, semantic segmentation, or instance segmentation.

[0054] Hereinafter, the image to be analyzed is also referred to as a "target image". The target image in the present embodiment is an image obtained by photographing the contour of a complex shape. As an example, the target image may also be an image obtained by photographing a plurality of particles in a dispersed state. As another example, the target image may also be an image obtained by photographing the surface of a metal precipitate.

[0055] Figure 1 : is a figure which shows an example of the target image in this embodiment. Figure 1 The target image 900 shown is an image obtained by photographing the surface of the metal precipitate. Figure 1 As shown, a plurality of contour lines having complex shapes are captured in the object image 900 .

[0056] It should be noted that Figure 1 The target image 900 shown in FIG. 1 is an image captured by a scanning electron microscope (SEM). Depending on the characteristics of the object to be captured, the purpose of image analysis, etc., the target image in this embodiment can be captured by, for example, an optical microscope or a transmission electron microscope (TEM).

[0057] Figure 2 This is a diagram showing an example of a teacher image in this embodiment. Figure 2 The teacher image 910 shown in FIG. Figure 1 The target image represented in is an image with contour lines drawn. Figure 2 In FIG. 9 , the area corresponding to the contour line is drawn with a thick line having a brightness different from the brightness of each pixel of the target image 900 .

[0058] In the past, the production of teacher images was performed manually by people using computers. Figure 1The contour of a complex shape like the target image 900 shown is difficult to display using a general input device such as a mouse or a keyboard, and therefore an input device dedicated to image editing such as a handwriting tablet is generally used.

[0059] When manually drawing Figure 2 When drawing contours of complex shapes as shown, for example, each contour may take several hours depending on the complexity of the image. For example, approximately hundreds of teacher images are required to train a machine learning model based on deep learning. Therefore, training a high-precision machine learning model requires a lot of labor and time.

[0060] In the production of the teacher image, it is not enough to simply draw all the contours captured in the image, but it is necessary to select and draw the necessary contours according to the purpose of image analysis. Therefore, it is difficult to create a general annotation tool that can correspond to any image and any analysis object.

[0061] Figure 3 is a diagram showing an example of a contour line image. The contour line image is an image obtained by extracting only the contour lines drawn on the teacher image. Figure 3 The contour line image 920 is obtained by Figure 1 Among the contour lines included in the target image 900 shown, only the contour lines that are to be analyzed are selected and drawn. Figure 3 In , the areas corresponding to the contours are drawn with white lines, and the other areas are drawn with black. Figure 3 As shown, in the contour image 920 , fine contours among the contours included in the target image 900 are omitted.

[0062] The teacher image production assistance system in this embodiment is intended to efficiently produce a teacher image for machine learning. In one aspect, according to this embodiment, the labor and time for producing a teacher image can be reduced.

[0063] <Overall structure of the teacher image production support system>

[0064] Will refer to Figure 4 The overall structure of the teacher image production support system in this embodiment is described. Figure 4 This is a block diagram showing an example of the overall structure of the teacher image production support system in this embodiment.

[0065] like Figure 4 As shown, the teacher image production assistance system 1 in this embodiment includes a model training device 10, an image processing device 20 and a user terminal 30. The model training device 10, the image processing device 20 and the user terminal 30 are connected to each other via a communication network N1 such as a LAN (Local Area Network) or the Internet for data communication.

[0066] The model training device 10 is an information processing device such as a personal computer, a workstation, or a server, which trains a machine learning model (hereinafter, also referred to as a "prediction model") for predicting contours contained in an image. The model training device 10 trains the prediction model based on a pre-made teacher image. The model training device 10 sends the trained prediction model to the image processing device 20.

[0067] The image processing device 20 is an information processing device such as a personal computer, a workstation, or a server that generates a teacher image based on a trained prediction model. The image processing device 20 receives a target image from a user terminal 30. The target image shows a position specified by the user (hereinafter, also referred to as a "target position"). The image processing device 20 predicts the contour line near the target position based on the received target image, and sends the teacher image showing the prediction result to the user terminal 30.

[0068] The user terminal 30 is an information processing terminal such as a personal computer, a tablet terminal, a smart phone, etc. operated by the user. The user terminal 30 accepts the user's designation of the target position and sends a target image representing the target position to the image processing device 20. The user terminal 30 receives a teacher image showing the prediction result of the contour line from the image processing device 20 and outputs it to the user.

[0069] The user of the teacher image production auxiliary system 1 inputs a target image to the user terminal 30 and performs an operation of specifying a target position where a contour line is desired to be drawn. The user terminal 30 outputs a teacher image showing a predicted result of a contour line near the position specified by the user to the user. The user can further perform an operation of specifying a target position on the output teacher image. In addition, the user can also perform an operation of editing the contour line shown in the teacher image. The user can produce a desired teacher image simply by performing an operation of selecting a target image in the target image.

[0070] It should be noted that Figure 4 The overall structure of the teacher image production auxiliary system 1 shown is only an example, and there may be various system structure examples according to the use and purpose. For example, one or more of the model training device 10, the image processing device 20, and the user terminal 30 may be provided in the teacher image production auxiliary system 1. For example, the model training device 10 or the image processing device 20 may be implemented by multiple computers or as a cloud computing service. Figure 4 The distinction between the model training device 10 , the image processing device 20 , the user terminal 30 and the like shown is merely an example.

[0071] <Hardware Structure of Teacher Image Production Assistance System>

[0072] Reference Figure 5The hardware structure of the teacher image production support system 1 in this embodiment is described.

[0073] Computer Hardware Structure

[0074] The model training device 10 , the image processing device 20 , and the user terminal 30 in this embodiment are implemented by, for example, a computer. Figure 5 This is a block diagram showing an example of the hardware configuration of the computer 500 in this embodiment.

[0075] like Figure 5 As shown, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The various hardware of the computer 500 are connected to each other via a bus 509. It should be noted that the input device 505 and the display device 506 can be connected to the external I / F 508.

[0076] The CPU 501 is a computing device that realizes the overall control and functions of the computer 500 by reading programs or data from a storage device such as the ROM 502 or the HDD 504 to the RAM 503 and executing processing.

[0077] ROM502 is an example of a nonvolatile semiconductor memory (storage device) that can retain programs or data even when the power is turned off. ROM502 is used as a main storage device that stores various programs, data, etc. required for CPU501 to execute various programs installed in HDD504. Specifically, ROM502 stores data such as boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface) that are executed when the computer 500 is started, OS (Operating System) settings, network settings, etc.

[0078] RAM 503 is an example of a volatile semiconductor memory (storage device) that deletes programs and data when the power is turned off. RAM 503 is, for example, DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), etc. RAM 503 provides a work area that is expanded when CPU 501 executes various programs installed in HDD 504.

[0079] The HDD 504 is an example of a nonvolatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software for controlling the entire computer 500, and application programs that provide various functions on the OS. In addition, the computer 500 may also use a storage device (e.g., an SSD (Solid State Drive)) that uses a flash memory as a storage medium instead of the HDD 504.

[0080] The input device 505 is a touch panel, operation keys or buttons, a keyboard or a mouse for a user to input various signals, a microphone for inputting sound data such as sound, or the like.

[0081] The display device 506 is constituted by a display such as liquid crystal or organic EL (Electro-Luminescence) for displaying a screen, a speaker for outputting audio data such as sound, and the like.

[0082] The communication I / F 507 is an interface for connecting the computer 500 to a communication network and performing data communication.

[0083] The external I / F 508 is an interface with external devices. The external devices include a driver device 510 and the like.

[0084] The drive device 510 is a device for setting the recording medium 511. The recording medium 511 includes a medium that records information optically, electrically, or magnetically, such as a CD-ROM, a floppy disk, a magneto-optical disk, etc. In addition, the recording medium 511 may include a semiconductor memory that records information electrically, such as a ROM, a flash memory, etc. Therefore, the computer 500 can read and / or write the recording medium 511 via the external I / F 508.

[0085] Furthermore, various programs installed on HDD 504 are installed, for example, by placing distributed recording medium 511 into drive device 510 connected to external I / F 508, and reading various programs recorded on recording medium 511 through drive device 510. Alternatively, various programs installed on HDD 504 may be installed by downloading from a network other than the communication network via communication I / F 507.

[0086] <Functional structure of teacher image production support system>

[0087] Reference Figure 6 The functional structure of the teacher image production support system in this embodiment is described. Figure 6 This is a block diagram showing an example of the functional configuration of the teaching image creation support system 1 according to the present embodiment.

[0088] 《Functional structure of model training device》

[0089] like Figure 6 As shown, the model training device 10 in this embodiment includes an image storage unit 100 , a training data generation unit 101 and a model training unit 102 .

[0090] The image storage unit 100 is composed of Figure 5 The HDD504 shown in the figure is implemented. The CPU501 executes Figure 5 The program expanded on the RAM 503 by the HDD 504 shown implements the training data generating unit 101 and the model training unit 102 .

[0091] Image data used for training the prediction model is stored in advance in the image storage unit 100. The image storage unit 100 may store one or more image data.

[0092] The image data includes a basic image and a contour image. The basic image is an image in which contours are captured. The contour image is an image showing the correct solution of contours contained in the basic image. The image data can be generated based on a pre-made teacher image. The basic image is an actual image that is the original image for making the teacher image. The contour image is an image obtained by extracting the area of ​​the contour from the teacher image.

[0093] The training data generation unit 101 generates training data for training the prediction model based on the image data read from the image storage unit 100. The amount of training data can be sufficient as long as it is sufficient to train the prediction model. The amount sufficient to train the prediction model can be determined based on the type of prediction model and the type of training algorithm.

[0094] The training data includes a training image and a correct image from which the correct contour line is extracted. The training image is an image showing a training position in an image in which the contour line is photographed. The so-called training position is a position that serves as a reference for predicting the relationship between a position in the model learning image and the contour line. The correct image from which the correct contour line is extracted is an image showing contour lines located near the training position among the contour lines contained in the training image.

[0095] The model training unit 102 trains the prediction model based on the training data generated by the training data generating unit 101. The model training unit 102 sends the trained prediction model to the image processing device 20.

[0096] The structure of the prediction model in this embodiment can use, for example, U-Net or various models derived from U-Net (for example, Nested U-Net, Attention U-Net, Swin U-Net, etc.).

[0097] 《Functional structure of image processing device》

[0098] like Figure 6 As shown, the image processing device 20 in this embodiment includes a model storage unit 200 , an image acquisition unit 201 , a contour line prediction unit 202 , and an image output unit 203 .

[0099] The model storage unit 200 is composed of Figure 5 The image acquisition unit 201, the contour line prediction unit 202 and the image output unit 203 are implemented by the CPU 501. Figure 5 The HDD 504 shown is implemented by processing of a program expanded on the RAM 503 .

[0100] The trained prediction model is stored in the model storage unit 200. The trained prediction model is generated by the model training device 10 and transmitted to the image processing device 20.

[0101] The image acquisition unit 201 acquires a target image showing a target position. The image acquisition unit 201 may acquire the target image by receiving the target image from the user terminal 30. The image acquisition unit 201 may acquire the target image by accepting an input of the target image via the input device 505 or the external I / F 508 of the image processing device 20.

[0102] The contour prediction unit 202 predicts the contour near the target position in the target image based on the trained prediction model read from the model storage unit 200. Specifically, the contour prediction unit 202 inputs the target image showing the target position into the trained prediction model, and obtains the prediction result output from the prediction model, which predicts the contour near the target position. The prediction result is two-dimensional data showing the probability of each pixel of the target image being a contour.

[0103] The image output unit 203 generates a teacher image by showing the target image the prediction result of the contour line prediction unit 202. The image output unit 203 transmits the teacher image to the user terminal 30.

[0104] 《Functional structure of user terminal》

[0105] like Figure 6 As shown, the user terminal 30 in this embodiment includes an image input unit 301 , an image display unit 302 , and a position specifying unit 303 .

[0106] By causing CPU501 to execute Figure 5 The program expanded on the RAM 503 by the HDD 504 shown implements the image input unit 301 , the image display unit 302 , and the position specifying unit 303 .

[0107] The image input unit 301 receives an input of a target image according to a user's operation and transmits the input target image to the image display unit 302 .

[0108] The image display section 302 displays the target image on the display device 506. When receiving the target image from the image input section 301, the image display section 302 displays the target image on the display device 506. When receiving the teacher image from the image processing device 20, the image display section 302 displays the teacher image on the display device 506 as a new target image.

[0109] The position specifying unit 303 receives, in accordance with a user operation, a designation of a target position for the target image displayed by the image display unit 302. The position specifying unit 303 displays the received target position on the target image, and transmits the target image showing the target position to the image processing device 20.

[0110] <Processing steps of teacher image production support system>

[0111] Reference Figures 7 to 14 The processing steps of the teaching image production support method executed by the teaching image production support system 1 in this embodiment will be described. Figure 7 This is a flowchart showing an example of the processing procedure of the teacher image production support method.

[0112] In step S1, the model training device 10 executes a training data generation process. The training data generation process is a process of generating training data for training a prediction model.

[0113] In step S2, the model training device 10 performs a model training process. The model training process is a process of training a prediction model using the training data generated in step S1.

[0114] In step S3, the image processing device 20 performs a teacher image generation process. The teacher image generation process is a process of generating a teacher image using the trained prediction model trained in step S2.

[0115] Training data generation and processing

[0116] Will refer to Figure 8The training data generation process in this embodiment is described in more detail. Figure 7 Step S1 in ). Figure 8 This is a flowchart showing an example of training data generation processing.

[0117] In step S11, the training data generation unit 101 of the model training device 10 reads the image data stored in the image storage unit 100. The training data generation unit 101 may read all the image data stored in the image storage unit 100. The training data generation unit 101 may also read one image data arbitrarily selected from the image data stored in the image storage unit 100.

[0118] In step S12, the training data generation unit 101 of the model training device 10 generates a training image based on the image data read in step S11. The training data generation unit 101 can generate the basic image itself contained in the image data as the training image. The training data generation unit 101 can generate a plurality of training images by randomly extracting images of a predetermined size from one basic image.

[0119] Fig. 9 is a diagram showing an example of a training image. Fig. 9 The training image 930 is obtained by Figure 1 An image obtained by extracting a portion of the target image 900 shown in FIG. Fig. 9 As shown, contours of complex shapes are captured in the training image 930 .

[0120] When the size of the image to be analyzed is large, if the prediction model is trained using the entire image, the amount of calculation during training may become huge. By using a portion of the image extracted from the large-size image to train the prediction model, the amount of calculation during training can be reduced.

[0121] return Figure 8 In step S13, the training data generation unit 101 of the model training device 10 generates a correct answer image based on the image data read in step S11. The training data generation unit 101 generates a correct answer image by extracting a region corresponding to the training image generated in step S12 from the contour line image.

[0122] When the basic image itself is used as the training image, the training data generating unit 101 can generate the contour line image itself as the correct answer image. When multiple training images are extracted from one basic image, the training data generating unit 101 can generate multiple correct answer images by extracting areas corresponding to each training image from the contour line image.

[0123] In order to create a teacher image for training a prediction model, it is necessary to draw contours manually. Manually drawing the contours of complex shapes requires a lot of labor and time. If multiple training data can be generated from one image data, the number of teacher images that must be prepared in advance can be reduced, and the prediction model can be trained efficiently as a whole.

[0124] Fig.10 This is a diagram showing an example of a correct answer image. Fig.10 The correct solution image 940 is obtained by Figure 3 An image obtained by extracting the same area as the training image 930 from the contour line image 920 shown in FIG. Fig.10 As shown, the correct image 940 shows the contour lines included in the training image 930 .

[0125] return Figure 8 In step S14, the training data generation unit 101 of the model training device 10 sets a training position in the training image generated in step S12. The training position is selected from the contour lines shown in the correct answer image generated in step S13.

[0126] For example, the training data generating unit 101 may randomly select a training position from the contour lines shown in the correct image. For another example, the training data generating unit 101 may select a training position from the contour lines shown in the correct image according to a predetermined rule.

[0127] For example, the training data generating unit 101 may indicate the training position in the training image by drawing an area representing a predetermined brightness at the training position. For another example, the training data generating unit 101 may indicate the training position in the training image by generating two-dimensional data indicating whether each pixel of the training image is a true value of the training position.

[0128] The brightness of the training position is drawn at a brightness different from the brightness included in the image. When the brightness that can be included in the image is limited, a brightness different from the brightness can be preset. When the brightness that can be included in the image is uncertain, the range of brightness included in the training image can be obtained, and the brightness not included in the range can be dynamically determined.

[0129] The number of training positions may be specified by the user or predetermined. The training data generating unit 101 may select the number of training positions specified by the user. The training data generating unit 101 may select the specified number of training positions.

[0130] Fig.11 FIG. 1 is a diagram showing an example of a training image to which a training position is added. Fig.11 The training image 950 shown is one in which Fig. 9In the training image 930 shown, images of three training positions 951 to 953 are provided.

[0131] return Figure 8 In step S15, the training data generation unit 101 of the model training device 10 extracts correct answer contour lines from the correct answer image generated in step S13. The correct answer contour lines are contour lines near the training position selected in step S14.

[0132] The vicinity range may be specified by the user or may be predetermined. That is, the training data generating unit 101 may accept the user's designation of the distance and extract the contour lines existing within the designated distance range with the training position as the center. The training data generating unit 101 may predetermine the distance of the specified vicinity range and extract the contour lines existing within the predetermined distance range with the training position as the center.

[0133] Fig.12 FIG. 1 is a diagram showing an example of a correct answer image from which a correct answer contour line is extracted. Fig.12 The correct solution image 960 is obtained from Fig.10 The image is obtained by extracting correct answer contour lines 961 to 963 near training positions 951 to 953 from the contour lines shown in the correct answer image 940 .

[0134] return Figure 8 In step S16, the training data generation unit 101 of the model training device 10 outputs the training data. The training data includes the training image obtained by showing the training position in step S14 and the correct answer image obtained by extracting the correct answer contour line in step S15.

[0135] 《Model Training Process》

[0136] Will refer to Fig.13 The model training process in this embodiment is described in more detail. Figure 7 Step S2). Fig.13 This is a flowchart showing an example of model training processing.

[0137] In step S21, the model training unit 102 of the model training device 10 receives the training data output from the training data generating unit 101. Next, the model training unit 102 selects a training image to be used for training from the received training data.

[0138] In step S22, the model training unit 102 of the model training device 10 inputs the training image selected in step S21 to the prediction model. The prediction model predicts the contour line near the training position shown in the training image, and outputs an image showing the prediction result.

[0139] In step S23, the model training unit 102 of the model training device 10 calculates the error between the correct image corresponding to the training image selected in step S21 and the prediction result predicted in step S22. The error can use, for example, L1 loss or cross entropy error. Then, the model training unit 102 updates the parameters of the prediction model to minimize the error. The update of the parameters can be performed according to a training algorithm corresponding to the structure of the prediction model.

[0140] In step S24, the model training unit 102 of the model training device 10 determines whether the error calculated in step S23 has converged (is small enough). Whether the error has converged can be determined based on a training algorithm corresponding to the type of prediction model.

[0141] When the error has not converged ("No"), the model training unit 102 returns the process to step S22. Thereafter, the model training unit 102 selects a different training image and performs the process from step S22 to step S24 again. When the error has converged ("Yes"), the model training unit 102 advances the process to step S25.

[0142] In step S25 , the model training unit 102 of the model training device 10 outputs the trained prediction model. The model training device 10 sends the trained prediction model output by the model training unit 102 to the image processing device 20 .

[0143] The image processing device 20 receives the trained prediction model from the model training device 10 . Next, the image processing device 20 stores the received trained prediction model in the model storage unit 200 .

[0144] Teacher Image Generation Processing

[0145] Reference Fig.14 The teacher image generation process in this embodiment is described in more detail. Figure 7 Step S3). Fig.14 It is a sequence diagram showing an example of the teacher image generation process.

[0146] In step S31, the image input unit 301 of the user terminal 30 receives input of a target image according to the user's operation. Then, the image input unit 301 transmits the input target image to the image display unit 302. The image display unit 302 receives the target image from the image input unit 301. Then, the image display unit 302 displays the received target image on the display device 506.

[0147] In step S32, the user specifies a target position where a contour line is desired to be drawn in the target image displayed on the display device 506 by operating the input device 505. The user can specify a plurality of target positions.

[0148] When a new target position is designated after continuing the operation of predicting the contour line in step S37 described later, the target position is designated by removing the predicted contour line.

[0149] The position specifying unit 303 of the user terminal 30 receives the designation of the target position according to the user's operation. Then, the position specifying unit 303 displays the target position on the target image. Then, the position specifying unit 303 transmits the target image showing the target position to the image processing device 20.

[0150] For example, the position specifying unit 303 may indicate the target position in the target image by drawing an area showing a prescribed brightness at the target position. For another example, the position specifying unit 303 may indicate the target position in the target image by associating two-dimensional data showing a true value of whether each pixel of the target image is the target position with the target image.

[0151] In addition, the position specifying unit 303 may automatically display the target position in the target image based on the brightness of the pixel without depending on the user's operation. In this case, the displayed target position may be selected according to the user's operation.

[0152] In step S33, the image acquisition unit 201 of the image processing device 20 receives the target image showing the target position from the user terminal 30. Next, the image acquisition unit 201 transmits the received target image to the contour line prediction unit 202 and the image output unit 203.

[0153] In step S34 , the contour line prediction unit 202 of the image processing device 20 receives the target image showing the target position from the image acquisition unit 201 . Next, the contour line prediction unit 202 reads the trained prediction model from the model storage unit 200 .

[0154] Next, the contour prediction unit 202 inputs the target image showing the target position into the trained prediction model. The prediction model predicts the contour near the target position in the target image and outputs an image showing the prediction result. Then, the contour prediction unit 202 transmits the prediction result to the image output unit 203.

[0155] In step S35, the image output unit 203 of the image processing device 20 receives the target image from the image acquisition unit 201. Then, the image output unit 203 receives the prediction result of the contour from the contour prediction unit 202.

[0156] Next, the image output unit 203 generates a teacher image by showing the prediction result of the contour line on the target image. Then, the image output unit 203 transmits the teacher image showing the prediction result of the contour line to the user terminal 30.

[0157] For example, the image output unit 203 may display the prediction result on the target image by drawing the contour line shown in the prediction result on the target image. For example, the image output unit 203 may display the prediction result on the target image by associating two-dimensional data showing the true value of whether each pixel of the target image is a contour line with the target image.

[0158] The brightness of the drawn contour line is a brightness different from the brightness included in the image. In addition, the brightness of the drawn contour line is a brightness different from the brightness of the drawn training position. The brightness of the drawn contour line can be predetermined or dynamically determined according to the brightness range included in the target image.

[0159] In step S36, the image display unit 302 of the user terminal 30 receives the teacher image from the image processing device 20. Next, the image display unit 302 displays the received teacher image on the display device 506 as a new target image.

[0160] In step S37, the user refers to the target image displayed on the display device 506 and determines whether to continue predicting the contour line. In the case of continuing to predict the contour line ("Yes"), the user performs an operation for continuing to predict the contour line. The operation for continuing to predict the contour line is, for example, an operation of specifying a new target position for the target image. If the user performs an operation for continuing to predict the contour line, the user terminal 30 returns the process to step S32.

[0161] If the contour line is not to be predicted ("No"), the user terminal 30 advances the process to step S38. Not to predict the contour line refers to the case where other operations other than the operation of specifying a new target position for the target image are performed. An example of other operations is an operation for editing the contour line, an operation for storing the teacher image, etc.

[0162] In step S38, the image display unit 302 of the user terminal 30 edits the contour line shown in the target image according to the user's operation for editing the contour line. The editing of the contour line includes the operation of deleting redundant contour lines, the operation of adding insufficient contour lines, etc. If the user does not perform the operation for editing the contour line, step S38 can be omitted.

[0163] After the operation for editing the contour line, the contour line may be continuously predicted. In the case of continuing to predict the contour line, if the user performs an operation for continuing to predict the contour line, the user terminal 30 returns the process to step S32.

[0164] In step S39 , the user terminal 30 stores the target image displayed on the display device 506 as a teacher image in a storage unit such as the HDD 504 according to the user's operation for storing the teacher image.

[0165] <Verification results>

[0166] Reference Figures 15 to 17 The results of verifying the prediction accuracy of the contour line using the prediction model in this embodiment will be described.

[0167] Fig.15 FIG. 1 is a diagram showing an example of a verification image. Fig.15 As shown, a contour line with a complex shape is captured in the verification image 970. Fig.15 In the verification image 970 shown, many target positions are drawn with a brightness different from the brightness of each pixel of the verification image 970 .

[0168] Fig.16 This is a diagram showing an example of a contour line image. Fig.16 The contour image 980 is shown from Fig.15 The image is obtained by extracting the contour line near the target position from the contour lines included in the verification image 970 shown. Fig.16 In the image 980 , the area corresponding to the contour line is drawn with high brightness, and the other areas are drawn with black. In other words, the contour line image 980 is an image showing an ideal prediction result for the verification image 970 .

[0169] Fig.17 This is a diagram showing an example of prediction results. Fig.17 The prediction result 990 shown is based on Fig.15 The prediction results output from the trained prediction model when the verification image 970 is input to the trained prediction model. Fig.17 In the figure, the area corresponding to the contour line shown in the prediction result is drawn in high brightness, and the other areas are drawn in black. Fig.17 As shown, in the prediction result 990 , the contour line near the target position selected in the verification image 970 can be predicted with high accuracy.

[0170] Figures 15 to 17 The verification results shown show that, by using the prediction model in this embodiment, a teaching image that accurately represents the contour line desired by the user can be efficiently created.

[0171] <Effects of Implementation Methods>

[0172] The image processing device 20 in this embodiment predicts the contour line near the target position in the target image based on the trained model that has learned the relationship between the position in the image and the contour line near the position. The user can draw the contour line only by specifying the desired position in the target image. In addition, since the contour line of the position not specified by the user is not drawn, the user can select and draw the contour line corresponding to the analysis object. Therefore, according to this embodiment, a teacher image for machine learning can be efficiently produced.

[0173] The image processing device 20 in this embodiment can repeatedly perform the acquisition of the target image showing the new target position and the prediction of the contour line near the new target position. The user can select the necessary contour line while confirming the prediction result. Therefore, according to this embodiment, the teacher image corresponding to the analysis object can be efficiently produced.

[0174] The model training device 10 in this embodiment can generate training data for training a prediction model by extracting a portion of an image from a pre-made teacher image. If the user prepares one teacher image, multiple training data can be generated. Therefore, according to this embodiment, a high-precision prediction model can be trained with less labor and time.

[0175] The model training device 10 of this embodiment can also randomly select a training position from the contour lines contained in the training image, and extract the contour lines near the training position from the correct image. The user does not need to specify the training position when training the prediction model. Therefore, according to this embodiment, training data for training the prediction model can be generated with fewer operations.

[0176] [Replenish]

[0177] Each function of the above-mentioned embodiments can be implemented by one or more processing circuits. Here, the "processing circuit" in this specification refers to a processor programmed to perform each function using software, such as a processor installed using an electronic circuit, an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and existing circuit modules and other devices designed to perform the above-mentioned functions.

[0178] Although the embodiments of the present invention have been described in detail, the embodiments of the present invention are illustrative and non-restrictive in all aspects. The embodiments may be modified and improved in various ways without departing from the scope and spirit of the appended claims. The matters recorded in the above-mentioned multiple embodiments may adopt other structures within the scope of non-contradiction, and may also be combined within the scope of non-contradiction.

[0179] This application claims the priority of Japanese Invention Patent Application No. 2023-66531 filed with the Japan Patent Office on April 14, 2023, the entire contents of which are incorporated herein by reference.

[0180] Description of Reference Numerals

[0181] 1 Teacher Image Production Assistance System

[0182] 10 Model training device

[0183] 100 Image storage unit

[0184] 101 Training data generation department

[0185] 102 Model Training Department

[0186] 20 Image processing device

[0187] 200 Model storage unit

[0188] 201 Image acquisition unit

[0189] 202 Contour Prediction Department

[0190] 203 Image output unit

[0191] 30 User Terminal

[0192] 301 Image input unit

[0193] 302 Image display unit

[0194] 303 Position specifying unit.

Claims

1. An image processing device, comprising: an image acquisition unit configured to acquire a target image showing a target position designated by a user; a contour line prediction unit configured to predict a contour line near the target position based on a trained model that has learned a relationship between a position in an image and a contour line near the position; as well as The image output unit is configured to output a teaching image for showing a prediction result of the contour line.

2. The image processing device according to claim 1, wherein: The image processing device repeatedly performs: acquiring, by the image acquisition unit, the target image showing the new target position; and The contour line prediction unit predicts the contour line near the new target position.

3. The image processing device according to claim 1 or 2, wherein: The trained model is a model that learns the relationship between the area and the contour line near the area based on training data including a training image that draws an area showing a specified brightness and a correct image that shows the contour line near the area contained in the training image.

4. The image processing device according to claim 3, wherein: The region is drawn at a position randomly selected from the contour lines contained in the training image.

5. The image processing device according to claim 3 or 4, wherein: The training data includes: the training image obtained by extracting a portion of a basic image including a contour line; and a correct answer image obtained by extracting a portion of the contour line image showing the contour line included in the basic image.

6. An image processing method, wherein: The computer performs the following steps: acquiring a target image showing a target location specified by a user; Predicting a contour line near the target location based on a trained model that learns a relationship between a location in an image and a contour line near the location; as well as A teacher image showing a prediction result of the contour line is output.

7. A program for causing a computer to execute the following steps: acquiring a target image showing a target location specified by a user; Predicting a contour line near the target location based on a trained model that learns a relationship between a location in an image and a contour line near the location; and A teacher image showing a prediction result of the contour line is output.

Images