Learning model creation method, learning model, estimation method, image processing system, and program

Abstract

To provide a learning model creation method, a learning model, an estimation method, an image processing system, and a program, which enable training of an efficient learning model capable of generating a large amount of learning data, to which a large amount of efficiently generated learning data is applied.SOLUTION: A learning model creation method is provided, comprising acquiring a region of normal objects included in a processing target image as a first mask (S10), generating a second mask by changing the state of the first mask (S12), and using the first mask and the second mask as learning data to train a learning model to estimate a difference with respect to the first mask from the second mask as a region of interest (S14), or to estimate the first mask from the second mask.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a method for producing a learning model, a learning model, an estimation method, an image processing system, and a program. [Background technology]

[0002] Segmentation is known as one of the image recognition techniques. Segmentation is the process of extracting one or more objects from an image. Progress is being made due to developments in machine learning models.

[0003] In segmentation learning models, masks are typically learned. A mask is understood as a region of interest, etc., extracted from an image. Since it is costly to create a mask, there are cases where a simulation image is used instead of a mask. On the other hand, when using simulation images, a domain gap occurs. There are challenges.

[0004] Patent Document 1 discloses a system for detecting changes in a product shelf from a photographed image of the product shelf. An image processing system is described that classifies the changes detected and assesses the condition of the merchandise shelves.

[0005] The device described in the document uses a foreground image and a background image in a photograph of a product shelf. Then, change areas are identified. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Patent No. 6911915 Summary of the Invention [Problem to be solved by the invention]

[0007] However, a large amount of correct answer data is required to train a segmentation model that extracts regions that deviate from normal regions as regions of interest, and creating such a large amount of correct answer data takes time.

[0008] The device described in Patent Document 1 detects changes in product shelves using a shelf change model that models changes in product shelves that have been learned in advance. The disclosure of the learning data for the shelf change model in this document does not include any description of the problem of acquiring the above-mentioned correct answer data.

[0009] The present invention has been made in consideration of the above circumstances, and aims to provide a method for manufacturing a learning model, a learning model, an estimation method, an image processing system, and a program that can efficiently generate a large amount of learning data and enable learning of a learning model to which the efficiently generated large amount of learning data is applied. [Means for solving the problem]

[0010] A manufacturing method for a learning model according to a first aspect of the present disclosure is a manufacturing method for a learning model that estimates a region of interest or estimates a normal region for an object included in an image based on a state change relative to a normal object having a specified state, in which a computer obtains a region of the normal object included in an image to be processed as a first mask, changes the state of the first mask to generate a second mask, and, using the first mask and the second mask as learning data, learns to estimate the difference between the second mask and the first mask as the region of interest, or learns to estimate the first mask from the second mask.

[0011] According to the manufacturing method of the learning model according to the first aspect, the first mask representing the normal region of the object is changed to generate the second mask, and pairs of the first mask and the second mask can be efficiently acquired as a large amount of learning data. This makes it possible to manufacture a learning model trained using pairs of the first mask and the second mask as learning data.

[0012] A method for manufacturing a learning model according to a second aspect may be such that, in the method for manufacturing a learning model according to the first aspect, the image to be processed is a medical image, and an anatomical structure is applied as the object.

[0013] According to this aspect, a learning model for use with medical images is produced.

[0014] A manufacturing method for a learning model according to a third aspect may be such that, in the manufacturing method for a learning model according to the second aspect, when generating a second mask, the shape of the first mask is deformed by combining the first mask with a shape that resembles a lesion.

[0015] According to this embodiment, a second mask based on the shape of the lesion can be obtained.

[0016] A manufacturing method for a learning model according to the fourth aspect may be such that, in the manufacturing method for a learning model according to the second aspect, when generating a second mask, shapes that are perceived as missing anatomical structures are omitted from the first mask.

[0017] According to this aspect, it is possible to obtain a second mask by deforming the first mask so as to imitate the absence of an anatomical structure.

[0018] A manufacturing method of a learning model according to the fifth aspect may be such that, in the manufacturing method of a learning model according to the first aspect, when generating a second mask, the shape of the first mask is expanded or the shape of the first mask is contracted.

[0019] According to this aspect, it is possible to obtain a second mask corresponding to an expansion of the normal region and a second mask corresponding to a contraction of the normal region.

[0020] A manufacturing method for a learning model according to a sixth aspect may be such that, in the manufacturing method for a learning model according to the first aspect, when generating a second mask, the shape of the first mask is deformed by combining the first mask with an abnormality simulation shape that simulates an abnormality.

[0021] According to this aspect, it is possible to obtain a second mask in which a simulated abnormality shape that simulates an abnormality is combined with the first mask.

[0022] A learning model manufacturing method according to a seventh aspect may be such that, in the learning model manufacturing method according to the sixth aspect, a plurality of abnormality simulation shapes are combined with the first mask when generating the second mask.

[0023] According to this embodiment, it is possible to handle the case where a single object has a plurality of abnormal eye regions.

[0024] A method for producing a learning model according to an eighth aspect may be such that, in the method for producing a learning model according to the sixth aspect, a rotated abnormality simulation shape is combined with the first mask when generating the second mask.

[0025] According to this embodiment, a plurality of second masks can be obtained from one simulated abnormal shape.

[0026] A manufacturing method for a learning model according to a ninth aspect may be such that, in the manufacturing method for a learning model according to the first aspect, when generating a second mask, an abnormality simulation shape that simulates an abnormality is omitted from the first mask, and the shape of the first mask is deformed.

[0027] According to this aspect, it is possible to deal with the case where there is an abnormal area that is recognized as a loss.

[0028] The learning model according to a tenth aspect of the present disclosure is a learning model that estimates a region of interest or a normal region for an object included in an image based on a state change relative to a normal object having a specified state, and uses a first mask representing a region of a normal object included in an image to be processed and a second mask generated by changing the state of the first mask as learning data, and is trained to estimate the difference from the second mask to the first mask as a region of interest, or to estimate the first mask from the second mask.

[0029] According to the learning model of the 10th aspect, it is possible to obtain the same effect as the manufacturing method of the learning model of the 1st aspect. The constituent elements of the manufacturing method of the learning model of the 2nd to 14th aspects can be applied to the constituent elements of the learning model of the other aspects.

[0030] An estimation method according to an eleventh aspect of the present disclosure is an estimation method for estimating a region of interest or a normal region for an object included in an image based on a state change relative to a normal object having a specified state, in which a computer extracts the region of the object included in the image to be processed as a third mask, and estimates from the third mask a region outside the region of the object in a normal state as a region of interest or estimates the normal region from the third mask.

[0031] According to the estimation method of the eleventh aspect, a learning model trained using a large amount of efficiently generated learning data can be applied to estimate an attention region or a normal region for an object included in an image.

[0032] An image processing system according to a twelfth aspect of the present disclosure is an image processing system that estimates a region of interest or a normal region for an object included in an image based on a state change relative to a normal object having a specified state, and includes one or more processors that execute instructions of a program to extract the region of the object included in the image to be processed as a third mask, and estimates from the third mask a region outside the region of the object in a normal state as a region of interest or estimates the normal region from the third mask.

[0033] According to the image processing system of the twelfth aspect, it is possible to obtain the same advantageous effects as the estimation method of the eleventh aspect.

[0034] A program according to a thirteenth aspect of the present disclosure is a program that estimates a region of interest or a normal region for an object included in an image based on a state change relative to a normal object having a specified state, and is a program that realizes a function of a computer to extract the region of the object included in the image to be processed as a third mask, and to estimate from the third mask a region outside the region of the object in a normal state as a region of interest, or to estimate the normal region from the third mask.

[0035] According to the program of the thirteenth aspect, it is possible to obtain the same advantageous effects as the estimation method of the eleventh aspect. Effect of the Invention

[0036] According to the present invention, a second mask is generated by changing a first mask that represents a normal region of an object, and pairs of the first mask and the second mask can be efficiently acquired as a large amount of learning data. This makes it possible to produce a learning model that is trained using pairs of the first mask and the second mask as learning data. [Brief description of the drawings]

[0037] [Figure 1]FIG. 1 is a flowchart showing the procedure of a method for producing a learning model according to an embodiment. [Diagram 2] FIG. 2 is a flow chart showing the procedure of the mask deformation processing step shown in FIG. [Diagram 3] FIG. 3 is a schematic diagram showing a specific example of a method for manufacturing a learning model according to an embodiment. [Figure 4] FIG. 4 is a functional block diagram showing the electrical configuration of the learning model producing system according to the embodiment. [Diagram 5] FIG. 5 is a block diagram illustrating an example of a hardware configuration of a learning model producing system according to an embodiment. [Figure 6] FIG. 6 is a flow chart showing the sequence of another embodiment of the mask deformation processing steps shown in FIG. [Figure 7] FIG. 7 is a flowchart showing the procedure of a method for producing a learning model according to a modified example. [Figure 8] FIG. 8 is a schematic diagram showing a specific example of obtaining the first mask. [Figure 9] FIG. 9 is a schematic diagram showing another specific example of obtaining the first mask. [Figure 10] FIG. 10 is a schematic diagram showing a specific example of a deformation mask. [Figure 11] FIG. 11 is a schematic diagram showing a first specific example of obtaining the second mask. [Figure 12] FIG. 12 is a schematic diagram showing a second specific example of obtaining the second mask. [Figure 13] FIG. 13 is a schematic diagram showing a third specific example of obtaining the second mask. [Figure 14] FIG. 14 is a schematic diagram showing a fourth specific example of obtaining the second mask. [Figure 15] FIG. 15 is a schematic diagram of the estimation model. [Figure 16] FIG. 16 is a schematic diagram of another example of the estimation model. [Figure 17] FIG. 17 is a flowchart showing the procedure of the estimation method according to the embodiment. [Figure 18] FIG. 18 is a conceptual diagram of the estimation method shown in FIG. [Figure 19] FIG. 19 is a functional block diagram showing the electrical configuration of the image processing system according to the embodiment. [Figure 20] FIG. 20 is a block diagram illustrating an example of a hardware configuration of the image processing system illustrated in FIG. [Figure 21] FIG. 21 is a schematic diagram showing an example of application to medical images. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0038] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In this specification, the same components are given the same reference numerals, and duplicated explanations will be omitted as appropriate.

[0039] [Outline of the manufacturing method for learning models] A method for manufacturing a learning model that performs segmentation will be described below. Here, an example of segmentation is given in which an abnormal area that deviates from a normal area is extracted as a region of interest from an image. The image to be processed may be a two-dimensional image or a three-dimensional image. An example of the image is a photographed image obtained by photographing a subject using a photographing device.

[0040] The manufacturing of the learning model may be understood as a term representing the construction of the learning model, such as the generation of the learning model and the creation of the learning model. Note that the abnormal region described in the embodiment is an example of a region of interest based on a state change for a normal object having a specified state.

[0041] The captured image may be a medical image of the anatomical structure of the human body, or an inspection image of an industrial product. The captured image may be an image of a structure such as a building, a road, or a bridge, or a natural image of a natural object such as a landscape. The term image may include the meaning of image data, which is a signal representing an image.

[0042] [Procedure for manufacturing learning models] FIG. 1 is a flowchart showing the flow of steps of a method for manufacturing a learning model according to an embodiment. In the method for manufacturing a learning model shown in the figure, each step is performed by a computer executing a program. The computer for executing each step included in the method for manufacturing a learning model may be one or more computers. The computer may be a virtual machine.

[0043] In the first mask acquisition step S10, a normal object included in the image is acquired as a first mask. A normal object is an object in a predetermined normal state. In the case of a medical image, normal anatomical structures such as normal organs, nerves, bones, and muscles are applied as normal objects.

[0044] In the first mask acquisition step S10, a plurality of first masks may be acquired from one image. The acquisition may include a mode in which a normal object is extracted from the image to be processed. The acquisition may include a mode in which a normal object previously extracted from the image to be processed is acquired.

[0045] The first mask acquisition step S10 may include a storage step of storing the acquired first mask. When the first mask is stored, identification information such as the name of the object may be added to each first mask. After the first mask acquisition step S10, the process proceeds to a first mask transformation process S12.

[0046] In the first mask transformation process S12, a part of the first mask acquired in the first mask acquisition process S10 is transformed to generate a second mask. In the first mask transformation process S12, a plurality of second masks different from each other are generated. Note that the first mask transformation process S12 described in the embodiment is an example of a process of generating a second mask by changing the state of the first mask.

[0047] Among the steps included in the manufacturing method of the learning model shown in Fig. 1, the steps including the first mask acquisition step S10 and the first mask transformation processing step S12 can be understood as steps of a learning data manufacturing method. After the first mask transformation processing step S12, the process proceeds to the learning step S14.

[0048] In the learning step S14, learning is performed to estimate a difference obtained by subtracting the first mask from the second mask, using a pair of the first mask acquired in the first mask acquisition step S10 and the second mask created in the first mask transformation processing step S12 as learning data. The learning model produced by executing each step from the first mask acquisition step S10 to the learning step S14 can function as an estimation model that estimates a region of interest from an image to be processed.

[0049] [Example of mask transformation processing] Fig. 2 is a flow chart showing the procedure of the mask deformation processing step shown in Fig. 1. In the mask deformation processing step, a deformation process of the first mask may be performed in which a deformed mask different from the first mask is superimposed on the first mask, and the first mask is regarded as having been deformed.

[0050] That is, the first mask transformation process S12 includes a transformed mask acquisition step S20 and a transformed mask synthesis step S22. In the transformed mask acquisition step S20, a transformed mask that is different from the first mask and that imitates an abnormal region that deviates from a normal region is acquired. Acquiring the transformed mask may include generating the transformed mask and correcting the transformed mask.

[0051] In the deformation mask acquisition step S20, a plurality of deformation masks different from each other are acquired. A plurality of deformation masks having similar shapes are regarded as a plurality of deformation masks different from each other. A deformation mask obtained by rotating an arbitrary deformation mask is regarded as a deformation mask different from the deformation mask before the rotation. After the deformation mask acquisition step S20, the process proceeds to the deformation mask synthesis step S22.

[0052] In the deformation mask synthesis step S22, the first mask is synthesized with each of the plurality of deformation masks acquired in the deformation mask acquisition step S20, to generate a plurality of second masks different from each other. An example of the synthesis here is a process of embedding the deformation mask in the first mask.

[0053] [Specific examples of manufacturing methods for learning models] 3 is a schematic diagram showing an overview of a method for producing a learning model according to an embodiment. The figure shows an example in which a normal kidney region in a medical image is acquired as a first mask 12, a sphere simulating a tumor is acquired as a deformed mask 14, and a second mask 16 is generated in which a part of the deformed mask 14 is exposed from the periphery of the first mask 12. The second mask 16 described in the embodiment is an example of an abnormality simulating shape simulating an abnormality.

[0054] FIG. 3 illustrates CNN as an example of a learning method for the learning model 10. CNN is an abbreviation of Convolution Neural Network, which is an English term that represents a convolutional neural network. CNN is a forward propagation type network that includes an input layer, a convolution layer, a pooling layer, a fully connected layer, and an output layer. CNN may include multiple convolution layers or multiple pooling layers.

[0055] 3, learning is performed using the first mask 12 and the second mask 16 as learning data, and a learning model 10 that estimates the difference 18 from the second mask 16 is generated. The learning model 10 is a learning model that is trained to output the difference 18 when the second mask 16 is input.

[0056] The learning model 10 may be a learning model that is trained to output the first mask 12 when the second mask 16 is input. The difference 18 corresponds to a portion of the second mask 16 that is exposed from the outer periphery of the first mask 12.

[0057] [Electrical configuration of the learning model manufacturing system] 4 is a functional block diagram showing the electrical configuration of a learning model production system according to an embodiment. The learning model production system 20 shown in the figure includes an image acquisition unit 22, a first mask acquisition unit 24, a transformed mask acquisition unit 26, a mask transformation processing unit 28, a learning unit 30, and a learning model storage unit 32.

[0058] The image acquisition unit 22 acquires an image to be processed. The image acquisition unit 22 may acquire the image to be processed from an image database. The image database may be an external device of the learning model production system 20 or an internal device.

[0059] The image acquisition unit 22 may acquire images from an imaging device that is communicably connected via a network. The imaging device may be an imaging device that acquires medical images, such as an endoscope device or an X-ray imaging device.

[0060] The first mask acquisition unit 24 acquires a normal object as the first mask 12 from an image acquired using the image acquisition unit 22. The first mask acquisition unit 24 executes the first mask acquisition step S10 shown in Fig. 1. The first mask acquisition unit 24 may apply an extraction model that is a trained learning model.

[0061] The deformation mask acquisition unit 26 acquires the deformation mask 14 shown in Fig. 3. The deformation mask acquisition unit 26 is the first mask deformation processing step S12 shown in Fig. 1, and executes the deformation mask acquisition step S20 shown in Fig. 2.

[0062] The mask transformation processing unit 28 synthesizes the first mask 12 acquired using the first mask acquisition unit 24 and the deformed mask 14 acquired using the deformed mask acquisition unit 26 to generate the second mask 16. The mask transformation processing unit 28 is the first mask transformation processing step S12 shown in Fig. 1, and executes the deformed mask synthesis step S22 shown in Fig. 2.

[0063] Of the learning model production system 20 shown in FIG. 4, a system including the image acquisition unit 22, the first mask acquisition unit 24, the deformed mask acquisition unit 26, and the mask transformation processing unit 28 can be understood as a learning data production system.

[0064] The learning unit 30 performs learning using a pair of a first mask 12 acquired using the first mask acquisition unit 24 and a second mask generated using the mask transformation processing unit 28 as learning data, and produces a learned learning model 10.

[0065] The learning model storage unit 32 stores the learning model 10 produced using the learning unit 30. The learning model storage unit 32 may be configured as an external device of the learning model production system 20 and configured to be able to freely communicate with the learning model production system 20 via a network.

[0066] 5 is a block diagram showing an example of a hardware configuration of the learning model manufacturing system according to the embodiment. The learning model manufacturing system 20 includes a processor 52, a non-transitory computer-readable medium 54, a communication interface 56, an input / output interface 58, and a bus 60.

[0067] The processor 52 includes a central processing unit (CPU) and may include a graphics processing unit (GPU). The processor 52 is connected to a computer-readable medium 54, a communication interface 56, and an input / output interface 58 via a bus 60.

[0068] The processor 52 reads various programs and data stored in the computer-readable medium 54 and executes various processes. The term "program" includes the concept of a program module and includes instructions equivalent to a program.

[0069] The computer-readable medium 54 is, for example, a storage device including a memory 62 which is a main storage device and a storage 64 which is an auxiliary storage device. The storage 64 is configured using, for example, a hard disk device, a solid-state drive device, an optical disk, a magneto-optical disk, a semiconductor memory, or the like. The storage 64 may be configured as an appropriate combination of the above-mentioned devices. The storage 64 stores various programs, data, and the like.

[0070] A hard disk device may be referred to as an HDD, which is an abbreviation of the English term Hard Disk Drive, and a solid state drive device may be referred to as an SSD, which is an abbreviation of the English term Solid State Drive.

[0071] The memory 62 includes an area used as a working area for the processor 52 and an area for temporarily storing programs and various data read from the storage 64. A program stored in the storage 64 is loaded into the memory 62, and the processor 52 executes instructions of the program, so that the processor 52 functions as a means for performing various processes defined by the program.

[0072] The memory 62 stores various programs, various data, and the like, such as a second mask acquisition program 70 and a learning program 72, which are executed by the processor 52. Each of the second mask acquisition program 70 and the learning program 72 may include a plurality of programs.

[0073] The memory 62 includes a first mask storage unit 74 in which the first mask 12 is stored, and a deformed mask storage unit 76 in which a deformed mask is stored. The processor 52 executes a second mask acquisition program 70, and generates a second mask 16 using the first mask 12 and the deformed mask 14. The processor 52 executes a learning program 72, and generates a learning model 10 using the first mask 12 and the second mask 16. The learning model 10 is stored in the storage 64. The word "generation" here may be included in the concept of acquisition.

[0074] The communication interface 56 performs communication processing with an external device by applying a wired or wireless connection, and exchanges information with the external device. The learning model production system 20 is connected to a communication line via the communication interface 56.

[0075] The communication line may be a local area network, a wide area network, or a combination of these. The communication line is not shown in the figure. The communication interface 56 can play the role of a data acquisition unit that accepts input of various data such as an original data set.

[0076] The learning model production system 20 includes an input device 66 and a display device 68. The input device 66 and the display device 68 are connected to the bus 60 via the input / output interface 58. The input device 66 may be, for example, a keyboard, a mouse, a multi-touch panel, other pointing devices, a voice input device, or the like. The input device 66 may be an appropriate combination of the above-mentioned keyboards, etc.

[0077] The display device 68 may be, for example, a liquid crystal display, an organic EL display, a projector, or the like. The display device 68 may be an appropriate combination of the above-mentioned liquid crystal displays, or the input device 66 and the display device 68 may be integrated together, such as a touch panel, or the information processing device of the learning model manufacturing system 20, the input device 66, and the display device 68 may be integrated together, such as a touch panel type tablet terminal. The organic EL display may be referred to as OEL, which is an abbreviation of organic electro-luminescence. The EL in the organic EL display is an abbreviation of Electro-Luminescence.

[0078] Here, examples of the hardware structure of the processor 52 include a CPU, a GPU, a PLD (Programmable Logic Device), and an ASIC (Application Specific Integrated Circuit). The CPU is a general-purpose processor that executes programs and acts as various functional units. The GPU is a processor specialized for image processing.

[0079] A PLD is a processor whose electrical circuitry can be reconfigured after the device is manufactured. An example of a PLD is the Field Programmable Gate Array (FPGA). An ASIC is a processor that contains dedicated electrical circuitry designed specifically to perform a specific task.

[0080] A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. Examples of combinations of various processors include a combination of one or more FPGAs and one or more CPUs, and a combination of one or more FPGAs and one or more GPUs. Another example of a combination of various processors includes a combination of one or more CPUs and one or more GPUs.

[0081] A single processor may be used to configure multiple functional units. An example of using a single processor to configure multiple functional units is a configuration in which a single processor is configured by applying a combination of one or more CPUs and software, such as a SoC (System On a Chip) as typified by a computer such as a client or server, and the processor is made to operate as multiple functional units.

[0082] Another example of using one processor to configure multiple functional units is a mode in which a processor is used to realize the functions of an entire system including multiple functional units using one IC chip.

[0083] In this way, the various functional units are configured as a hardware structure using one or more of the various processors described above. More specifically, the hardware structure of the various processors described above is an electric circuit (circuitry) that combines circuit elements such as semiconductor elements.

[0084] [Variations of the manufacturing method of the learning model] Fig. 6 is a flow chart showing the sequence of another aspect of the mask deformation processing step shown in Fig. 1. In the mask deformation processing step shown in Fig. 6, a deformation process of the first mask can be carried out to directly edit the first mask.

[0085] That is, the mask transformation process includes an editing position designation step S30 and an editing method designation step S32. In the editing position designation step S30, an arbitrary point on a closed curve that represents the periphery of the first mask is designated.

[0086] In the editing method designation step S32, an editing method for the editing point designated in the editing position designation step S30 is designated. For example, the editing point is moved, and a smoothing process is performed on the line segments around the moved editing point.

[0087] In the editing method designation step S32, an editing method for the first mask that expands the first mask may be designated, thereby partially expanding the first mask, or an editing method for the first mask that removes a portion of the first mask may be designated, thereby partially contracting the first mask.

[0088] In the mask transformation processing step shown in Fig. 6, different editing methods may be applied to the same editing target point to generate a plurality of second masks that are different from each other. Also, in the mask transformation processing step shown in Fig. 6, the same editing method or different editing methods may be applied to a plurality of editing target points to generate a plurality of second masks that are different from each other.

[0089] 7 is a flowchart showing the procedure of a method for producing a learning model according to a modified example. The flowchart shown in Fig. 7 includes a learning step S15 instead of the learning step S14 in the flowchart shown in Fig. 1.

[0090] In the learning step S15 shown in Fig. 7, a pair of the first mask and the second mask is used as learning data, and learning is performed to estimate a normal region, which is a region corresponding to the first mask, from the second mask. The learned learning model generated by applying the learning model manufacturing method shown in Fig. 7 is stored in the learning model storage unit 32 shown in Fig. 4.

[0091] [Example of the first mask] As a specific example of the first mask, an example will be shown in which a normal anatomical structure is extracted from a medical image and the extracted normal anatomical structure is acquired as the first mask. Here, the kidney is taken as an example of the anatomical structure.

[0092] Fig. 8 is a schematic diagram showing a specific example of acquiring the first mask. Fig. 8 shows an example in which a kidney region A1 is extracted from a coronal cross-sectional image P1 of a normal kidney, and the kidney region A1 is acquired as a first mask 12A.

[0093] Fig. 9 is a schematic diagram showing another specific example of acquiring the first mask. Fig. 9 shows an example in which a kidney region A2 is extracted from an axial slice image P2 of a normal kidney, and the kidney region A2 is acquired as a first mask 12B.

[0094] In this manner, a large number of medical images including normal organs are prepared, normal organs are extracted from each of the medical images, and a large number of first masks are obtained. Fig. 8 illustrates a coronal slice image P1, and Fig. 9 illustrates an axial slice image P2, but the medical image including normal organs may be a sagittal slice image. In addition, the medical image including normal organs may be a three-dimensional image.

[0095] [Example of a deformation mask] Fig. 10 is a schematic diagram showing a specific example of a deformation mask. Fig. 10 shows an example in which a deformation mask imitating a lesion such as a tumor is acquired as the deformation mask 14. Fig. 10 shows three types of deformation masks 14A, 14B, and 14C having different shapes.

[0096] Instead of a deformation mask imitating a tumor, an actual tumor extracted from a medical image may be acquired as the deformation mask. Although the deformation mask 14A imitating a tumor is illustrated in Fig. 10, the deformation mask may imitate the shape of a polyp or the like.

[0097] [Example of the second mask] Fig. 11 is a schematic diagram showing a first specific example of obtaining a second mask. Fig. 11 shows an example in which two different types of second masks 16A and 16B are obtained by combining the first mask 12A shown in Fig. 8 and the deformed mask 14A shown in Fig. 10. The second masks 16A and 16B shown in Fig. 11 have the same first mask 12A and deformed mask 14A, but differ in the position of the first mask 12A where the deformed mask 14A is superimposed.

[0098] FIG. 11 illustrates second masks 16A and 16B in which a portion of deformed mask 14A protrudes from first mask 12A and first mask 12A is expanded, but it is also possible to obtain a second mask in which a portion of deformed mask 14A is missing from first mask 12A.

[0099] For example, the entire deformed mask 14A may be deleted from the second mask 16A in which the deformed mask 14A is superimposed on the first mask 12A to obtain a second mask in which a part of the first mask 12A is missing.

[0100] Fig. 12 is a schematic diagram showing a second specific example of obtaining a second mask. The second mask 16C shown in the drawing is obtained by combining the first mask 12A shown in Fig. 8 with the deformed mask 14A and the deformed mask 14B shown in Fig. 10 to obtain one type of second mask 16C.

[0101] 12, two different types of deformed masks, 14A and 14B, are superimposed on the first mask 12A in the second mask 16C. The position where the deformed mask 14A is superimposed on the first mask 12A is different from the position where the deformed mask 14B is superimposed on the first mask 12A.

[0102] Fig. 13 is a schematic diagram showing a third specific example of obtaining a second mask 16D, which is obtained by combining the first mask 12B shown in Fig. 9 and the modified mask 14B shown in Fig. 10.

[0103] Fig. 14 is a schematic diagram showing a fourth specific example of obtaining a second mask. The second mask 16E shown in the figure is obtained by combining the first mask 12B shown in Fig. 9 and the deformed mask 14B shown in Fig. 10. The second mask 16E shown in Fig. 14 is different from the second mask 16D shown in Fig. 13 in the position of the first mask 12B on which the second mask 16E is superimposed.

[0104] That is, the second mask 16A etc. is obtained by randomly combining one or more of the multiple deformed masks 14A etc. for each of the multiple first masks 12A etc. The deformed mask 14A etc. to be superimposed on the first mask 12A etc. may be disposed at a position that protrudes from the first mask 12A etc. If the area of ​​the deformed mask 14A etc. that protrudes from the first mask 12A etc. is too small, it becomes difficult to detect the difference between the first mask 12A etc. and the second mask 16A etc., so a minimum value of the area of ​​the deformed mask 14A etc. that protrudes from the first mask 12A etc. may be specified.

[0105] Further, the second masks 16A etc. are arranged such that the deformed masks 14A etc. do not cover the first masks 12A etc. too much.

[0106] [Specific examples of learning models] Fig. 15 is a schematic diagram of an estimation model. Fig. 15 shows an estimation model 11 to which the trained learning model 10 shown in Fig. 3, which is manufactured using the above-mentioned manufacturing method for a learning model, is applied. The estimation model 11 is manufactured by performing learning such that a difference 18 between the first mask 12 and the second mask 16 is estimated using a pair of a first mask 12, which is a region in a normal state, and a second mask 16, which includes an abnormal region that deviates from the normal state, as learning data.

[0107] The estimation model 11 extracts an object from the image to be processed as a third mask 106, and estimates an abnormal region of the third mask 106, which is understood as the difference between the normal region 102 and the third mask 106, as a region of interest 108 from the third mask 106.

[0108] Fig. 16 is a schematic diagram of another example of an estimation model. The estimation model 11A shown in Fig. 16 is produced by performing learning to estimate the first mask 12 using a pair of a first mask 12 representing a normal region and a second mask 16 including an abnormal region as learning data. The estimation model 11A extracts an object region from the image to be processed as a third mask 106, and estimates a normal region 102, which is a region in a normal state, from the third mask 106.

[0109] [Effects of the embodiment] The learning model manufacturing method and learning model manufacturing system according to the embodiment can provide the following advantageous effects.

[0110] [1] One or more first masks 12 representing normal objects are obtained from an image including the normal objects. The first masks are transformed to obtain a plurality of second masks 16. In this way, a large amount of learning data is obtained.

[0111] Using a pair of the first mask 12 and the second mask 16 as learning data, a difference between the first mask 12 and the second mask 16 is estimated, or learning is performed to estimate the first mask 12, and a learned learning model is manufactured. In this way, an estimation model that estimates a region that deviates from a normal region can be manufactured using a large amount of learning data. In addition, it is possible to reduce the dependency of the estimation model on the texture of the image.

[0112] [2] A deformed mask 14 that imitates an abnormal region that deviates from a normal region is obtained. The deformed mask 14 is superimposed on the first mask 12, and the first mask 12 is deformed so that a part of the deformed mask 14 protrudes from the first mask 12, thereby generating a second mask 16. This makes it possible to obtain a large number of second masks 16 that have regions similar to actual abnormal regions.

[0113] [3] A plurality of modified masks 14 are combined with one first mask 12. In this way, a large number of second masks 16 can be obtained from one first mask 12.

[0114] [4] A single deformation mask 14 is rotated to obtain a plurality of deformation masks 14. In this way, a large number of deformation masks 14 can be obtained from a single deformation mask 14.

[0115] [Procedure of Estimation Method According to the Embodiment] Fig. 17 is a flowchart showing the procedure of an estimation method according to an embodiment. The estimation method shown in the figure estimates an abnormal area that deviates from a normal area of ​​an object from an image including the object. The estimation method shown in the figure can be implemented using the trained learning model 10 shown in Fig. 4.

[0116] That is, in the estimation method shown in Fig. 17, each step is executed by a computer executing a program. The number of computers that execute each step included in the estimation method may be one or more. The computer may be a virtual machine.

[0117] In the third mask extraction step S100, a region of an object is extracted as a third mask from the image to be processed. The third mask extraction step S100 may apply a known extraction model.

[0118] The image to be processed may be a two-dimensional image or a three-dimensional image. An example of the image is a photographed image obtained by photographing a subject using a photographing device. The photographed image may be a medical image of the anatomical structure of the human body, or an inspection image of an industrial product. The photographed image may be an image of a building, road, bridge, or other structure, or a natural image of a natural object such as a landscape.

[0119] The object may be an anatomical structure of a human body in a medical image, or a part of an industrial product to be inspected. The object may be a wall of a building, a pier of a bridge, a road, etc. After the third mask extraction step S100, the method proceeds to an abnormality estimation step S102.

[0120] In the abnormality estimation step S102, an abnormal region in the third mask extracted in the third mask extraction step S100 is estimated. The abnormal region is a region that deviates from a normal region of the object extracted as the third mask. After the abnormality estimation step S102, an abnormality region storage step may be executed to store the extracted abnormal region. The abnormality region storage step may be included in the abnormality estimation step S102.

[0121] The estimation model 11A shown in FIG. 16 may implement an estimation method including a normality estimation step in which a normal region in a third mask is estimated, instead of the abnormality estimation step S102 shown in FIG.

[0122] [Summary of the estimation model] Fig. 18 is a conceptual diagram of the estimation method shown in Fig. 17. In the third mask extraction step S100 shown in Fig. 17, an object region A11 is extracted as a third mask 106 from an image P11 to be processed.

[0123] The estimation model 11 is a trained learning model that has been trained to estimate an abnormal region as an attention region 108 from the third mask 106. In the abnormality estimation step S102 shown in Fig. 17, when the third mask 106 is input to the estimation model 11, the attention region 108 is estimated from the third mask 106. The trained learning model 10 shown in Fig. 4 is applied to the estimation model 11 shown in Fig. 18.

[0124] [Image processing system configuration example] 19 is a functional block diagram showing the electrical configuration of an image processing system according to an embodiment of the present invention, in which a computer that executes the estimation method shown in FIG.

[0125] The image processing system 110 includes an image acquisition unit 112, a third mask extraction unit 114, an estimation unit 116, and an estimation result storage unit 118. The image acquisition unit 112 acquires an image to be processed, which is applied to the third mask extraction step S100 shown in FIG.

[0126] The third mask extraction unit 114 executes the third mask extraction step S100 shown in FIG.

[0127] The estimation unit 116 executes the abnormality estimation step S102 shown in Fig. 17 using the estimation model 11 shown in Fig. 18 to estimate the region of interest 108 from the third mask 106 acquired using the third mask extraction unit 114. The region of interest 108 estimated using the estimation unit 116 is stored in the estimation result storage unit 118 as an estimation result of the region of interest 108 in the processing target image. The estimation unit 116 may function as a processing unit that estimates a normal region from the third mask 106 using the estimation model 11A shown in Fig. 16.

[0128] Fig. 20 is a block diagram showing an example of a hardware configuration of the image processing system shown in Fig. 19. The processor 122, the communication interface 126, and the input / output interface 128 shown in Fig. 20 are similar to the processor 52, the communication interface 56, and the input / output interface 58 shown in Fig. 5, respectively. The input device 136 and the display device 138 shown in Fig. 20 are similar to the input device 66 and the display device 68 shown in Fig. 5, respectively. Here, the description of the processor 122 etc. will be omitted as appropriate.

[0129] 5 in that it is a storage device including a memory 132 serving as a main storage device and a storage 134 serving as an auxiliary storage device. The memory 132 and the storage 134 may each have the same configuration as the memory 62 and the storage 64 shown in FIG.

[0130] The memory 132 stores a third mask extraction program 140 executed by the processor 122. The memory 132 stores an estimation program 142 as the estimation model 11.

[0131] The third mask extraction program 140 is applied to the third mask extraction unit 114 shown in Fig. 19 to realize a third mask extraction function. The estimation program 142 is applied to the estimation unit 116 to realize an estimation function.

[0132] [Application to medical images] Fig. 21 is a schematic diagram showing an example of application to a medical image. Deformation of organs such as the kidney, thyroid gland, liver, and pancreas may include partial atrophy. Fig. 21 illustrates an estimation model 11 in which a third mask 106A representing an organ as an object is extracted from a medical image, and an abnormal region missing from a normal region is estimated as an attention region 108A of the third mask 106A.

[0133] The learning data applied to learning the estimation model 11 may be a first mask 12 representing a normal organ and a second mask 16 obtained by superimposing a deformed mask 14 representing partial atrophy on the first mask 12 and deleting from the first mask 12 the portion where the deformed mask 14 is superimposed on the first mask 12.

[0134] [Effects of the estimation method and image processing system according to the embodiment] The estimation method and image processing system according to the embodiment can provide the following advantageous effects.

[0135] [1] Using an estimation model trained to estimate a difference 18 between training data of a first mask 12 representing a normal region and a second mask 16 that is deformed to imitate an abnormal region relative to the normal region and the second mask 16, a region of interest 108 that is an abnormal region is estimated from a third mask 106 that is a region of an object extracted from the processing target image P11. This allows for highly accurate estimation of the region of interest 108 for the processing target image P11.

[0136] [2] Using a pair of the first mask 12 and the second mask 16 as learning data, an estimation model trained to estimate the first mask from the second mask 16 is used to estimate a normal region from the third mask 106, which is a region of an object extracted from the processing target image P11. This allows highly accurate estimation of a normal region for the processing target image P11.

[0137] [3] The expansion or contraction of the object is estimated as the abnormal region of the object. In this way, when the deformation of the object is the expansion or contraction of the object, the abnormal region of the object can be estimated.

[0138] The technical scope of the present invention is not limited to the scope described in the above embodiments. The configurations and the like in each embodiment can be appropriately combined with each other without departing from the spirit of the present invention. [Explanation of symbols]

[0139] 10 Learning Model 11 Estimation model 11A Estimation Model 12 First Mask 12A First Mask 12B First Mask 14 Second Mask 14A Deformed Mask 14B Transformation Mask 14C Transformation Mask 16 Second Mask 16A Second Mask 16B 2nd Mask 16C 2nd Mask 16D 2nd Mask 16E 2nd Mask 18 Difference 20 Learning Model Manufacturing System 22 Image acquisition section 24 First Mask Acquisition Section 26 Deformation mask acquisition unit 28 Mask transformation processing section 30 Learning Department 32 Learning model memory unit 52 processors 54 Computer-readable media 56 Communication Interface 58 Input / Output Interface 60 Bus 62 Memory 64 Storage 66 Input Devices 68 Display device 70 2nd Mask Acquisition Program 72 Study Programs 74 First mask memory unit 76 Transformation mask memory section 100 Information processing device 102 Normal area 106 3rd Mask 106A 3rd Mask 108 Areas of Interest 108A Area of ​​Interest 110 Image Processing System 112 Image acquisition unit 114 Third mask extraction unit 116 Estimation Department 118 Estimation result storage unit 122 processors 124 Computer-Readable Medium 126 Communication Interface 128 I / O Interface 132 Memory 134 and storage 136 Input Devices 138 Display Device 140 Third Mask Extraction Program 142 Estimation Program A1 Kidney Disease A2 Kidney area A11 Area of ​​object P1 Coronal section image P2 Axial cross-sectional image P11 Image to be processed S10 to S15: Each step in the manufacturing method of the learning model S20 to S32 Each step of the first mask deformation process S100 to S102: Each step of the estimation method

Claims

1. A method for manufacturing a learning model that estimates an attention area or a normal area for an object included in an image based on a state change with respect to a normal object having a specified state, The computer A region of a normal object included in the image to be processed is obtained as a first mask; Varying the state of the first mask to generate a second mask; A method for manufacturing a learning model that uses the first mask and the second mask as learning data and learns to estimate the difference between the second mask and the first mask as the region of interest, or learns to estimate the first mask from the second mask.

2. The image to be processed is a medical image, The method for producing a learning model according to claim 1 , wherein an anatomical structure is applied as the object.

3. The method for producing a learning model according to claim 2 , wherein when generating the second mask, the shape of the first mask is deformed by combining the first mask with a shape that resembles a lesion.

4. The method for producing a learning model according to claim 2 , wherein when generating the second mask, shapes that are recognized as missing parts of the anatomical structure are omitted from the first mask.

5. The method for producing a learning model according to claim 1 , wherein when generating the second mask, the shape of the first mask is expanded or contracted.

6. The method for manufacturing a learning model according to claim 1 , wherein when generating the second mask, a simulated abnormality shape that simulates an abnormality is combined with the first mask to deform the shape of the first mask.

7. The method for producing a learning model according to claim 6 , wherein a plurality of the simulated abnormal shapes are combined with the first mask when the second mask is generated.

8. The method for producing a learning model according to claim 6 , wherein when generating the second mask, the rotated abnormality simulation shape is combined with the first mask.

9. The method for manufacturing a learning model according to claim 1 , wherein when generating the second mask, an abnormality simulation shape that simulates an abnormality is removed from the first mask, and the shape of the first mask is deformed.

10. A learning model that estimates an attention area or a normal area for an object included in an image based on a state change with respect to a normal object having a specified state, A learning model that uses a first mask representing a region of a normal object included in an image to be processed and a second mask generated by changing the state of the first mask as learning data, and is trained to estimate the difference between the second mask and the first mask as the region of interest, or to estimate the first mask from the second mask.

11. An estimation method for estimating a region of interest or a normal region of an object included in an image based on a state change with respect to a normal object having a specified state, comprising: The computer extracting an object region included in the image to be processed as a third mask; An estimation method for estimating a region outside the region of the normal object from the third mask as the region of interest, or for estimating the normal region from the third mask.

12. An image processing system that estimates a region of interest or a normal region of an object included in an image based on a state change relative to a normal object having a specified state, one or more processors; The one or more processors execute instructions of a program to extracting an object region included in the image to be processed as a third mask; An image processing system that estimates a region outside the region of the normal object from the third mask as the region of interest, or estimates the normal region from the third mask.

13. A program for estimating a region of interest or a normal region of an object included in an image based on a state change relative to a normal object having a specified state, The computer A function of extracting an object region included in the image to be processed as a third mask; and A program that realizes a function of estimating an area outside the area of ​​the normal object from the third mask as the area of ​​interest, or estimating the normal area from the third mask.

Images