Image processing device, image processing method, and image processing program

Abstract

The purpose of the present invention is to improve segmentation accuracy when region segmentation is performed on a heterogeneous target group image. The image processing apparatus includes: a binary image generation unit that generates a binary image from a target image using a first learning completion model; a prediction unit that predicts a contour line of a target region in a binary image generated from the target image using a second learning completion model; an expansion contour line image generation unit for performing expansion processing on the predicted contour line to generate an expansion contour line image; a foreground region image generation unit that generates a foreground region image on the basis of a binary image generated from the target image and the expanded contour line image; and a region segmentation unit that segments the target image for each target region by a watershed method using background pixels determined on the basis of a binary image generated from the target image, boundary pixels determined on the basis of the expanded contour line image, and foreground pixels determined on the basis of the foreground region image.

Description

Technical Field

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program. Background Art

[0002] As a region segmentation method for determining the region of a target included in a target image and segmenting the target image according to the determined region of the target, for example, there can be cited methods using image analysis such as the watershed method, or methods using deep learning such as the R-CNN and Selective Search method.

[0003] <Prior Art Literature>

[0004] <Patent Documents>

[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-120815 Summary of the invention

[0006] <Problems to be Solved by the Invention>

[0007] On the other hand, when an image with a dense concentration of multiple inhomogeneous targets (object groups) (for example, an image obtained by photographing a group of particles with various internal depressions and protrusions or shades, and different shapes and sizes) is used as the object image, it is difficult to achieve high-precision region segmentation.

[0008] For example, in the case of the above-mentioned method using image analysis, it is necessary to find a parameter suitable for any one of a plurality of targets. In addition, in the case of the above-mentioned method using deep learning, it is necessary to prepare a huge number of images including objects of various forms as learning data.

[0009] The object of the present invention is to improve the segmentation accuracy when performing region segmentation on an image of a heterogeneous target group.

[0010] <Methods used to solve the problem>

[0011] According to a first aspect of the present disclosure, there is provided an image processing device, comprising: a binary image generating unit, which generates a binary image from an object image using a first learned model that binarizes the object image into a target area and a background area and generates a binary image; a prediction unit, which predicts the contour of the target area in the binary image generated from the object image using a second learned model that predicts the contour of the target area in the binary image; a dilated contour image generating unit, which dilates the predicted contour to generate a dilated contour image; a foreground area image generating unit, which generates a foreground area image based on the binary image generated from the object image and the dilated contour image; and a region segmentation unit, which segments the object image into each target area by a watershed method using background pixels determined based on the binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground area image.

[0012] A second aspect of the present disclosure is the image processing device according to the first aspect, wherein the binary image generation unit generates the binary image from the target image using a learned U-Net as the first learned model.

[0013] A third aspect of the present disclosure is the image processing device according to the second aspect, wherein the prediction unit uses a learned U-Net as the second learned model to generate a contour line image from a binary image generated from the object image.

[0014] A fourth aspect of the present disclosure is the image processing device according to the third aspect, wherein the foreground area image generation unit generates the foreground area image by calculating a difference between a binary image generated from the object image and the dilated contour line image.

[0015] A fifth aspect of the present disclosure is the image processing device according to the first aspect, wherein the target image is an image obtained by imaging a particle group under predetermined conditions.

[0016] A sixth aspect of the present disclosure is the image processing device according to the fifth aspect, comprising: a statistical processing unit that analyzes and performs statistical processing on an image obtained by dividing the object image into each target area.

[0017] A seventh aspect of the present disclosure is the image processing device according to the sixth aspect, wherein the statistical processing unit changes process conditions of a process for generating the particle group based on a result of statistical processing.

[0018] Aspect 8 of the present disclosure provides an image processing method, in which a computer executes the following processes: a process of generating a binary image from an object image using a first learned model that binarizes the object image into a target area and a background area and generates a binary image; a process of predicting the contour of the target area in the binary image generated from the object image using a second learned model that predicts the contour of the target area in the binary image; a process of dilating the predicted contour to generate a dilated contour image; a process of generating a foreground area image based on the binary image generated from the object image and the dilated contour image; and a process of segmenting the object image according to each target area by a watershed method using background pixels determined based on the binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground area image.

[0019] A ninth aspect of the present disclosure provides an image processing program for causing a computer to execute the following processes: a process of generating a binary image from an object image using a first learned model that binarizes the object image into a target area and a background area and generates a binary image; a process of predicting the contour of the target area in the binary image generated from the object image using a second learned model that predicts the contour of the target area in the binary image; a process of dilating the predicted contour to generate a dilated contour image; a process of generating a foreground area image based on the binary image generated from the object image and the dilated contour image; and a process of segmenting the object image according to each target area by a watershed method using background pixels determined based on the binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground area image.

[0020] <Effects of the Invention>

[0021] According to the present disclosure, it is possible to improve the segmentation accuracy when performing region segmentation on an image of a heterogeneous target group. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] [ Figure 1 ] Figure 1 It is a diagram showing an example of the system configuration of the process control system.

[0023] [ Figure 2 ] Figure 2 is a diagram showing an example of the hardware configuration of the image processing apparatus.

[0024] [ Figure 3 ] Figure 3 is a diagram showing a specific example of a process control system.

[0025] [ Figure 4 ] Figure 4 A diagram for explaining the details of a target image.

[0026] [ Figure 5 ] Figure 5 This is a diagram showing an example of the functional configuration of a region segmentation unit when a general watershed method is used.

[0027] [ Figure 6 ] Figure 6 FIG. 1 is a first diagram showing a specific example of processing performed by the region segmentation unit when a general watershed method is used.

[0028] [ Figure 7 ] Figure 7 FIG. 2 is a second diagram showing a specific example of processing performed by the region segmentation unit when a general watershed method is used.

[0029] [ Figure 8 ] Figure 8 FIG. 1 is a diagram showing an example of the functional configuration of the area dividing unit.

[0030] [ Fig. 9 ] Fig. 9 FIG. 1 is a first diagram showing a specific example of processing performed by the area segmentation unit.

[0031] [ Fig.10 ] Fig.10 FIG. 2 is a second diagram showing a specific example of the processing performed by the area segmentation unit.

[0032] [ Fig.11 ] Fig.11 This is a diagram showing an example of the functional configuration of the statistical processing unit.

[0033] [ Fig.12 ] Fig.12 : is a flowchart showing the flow of the process control processing.

[0034] [ Fig.13 ] Fig.13 : is a flowchart showing the flow of the area segmentation process. DETAILED DESCRIPTION

[0035] Hereinafter, each embodiment will be described with reference to the drawings. It should be noted that in this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals to omit repeated descriptions.

[0036] [First embodiment]

[0037] <System Configuration of Process Control System>

[0038] First, the overall system configuration of a process control system including an image processing device according to the first embodiment will be described. Figure 1 It is a diagram showing an example of the system configuration of the process control system.

[0039] like Figure 1 As shown, the process control system 100 includes a target process 110 , a target image generating process, and an image processing device 130 .

[0040] The target process 110 is a process for generating a plurality of particles (particle groups), that is, a particle group generating process. The particle group mentioned here includes, for example, a sintered body group generated by a sintering process or a spheroid generated by a culture process.

[0041] It should be noted that the particle group generated by the target process 110 is sent to the target image generation process 120 .

[0042] The target image generation process 120 generates an image in which a plurality of targets (target groups) are densely packed by imaging the transported particle group under predetermined conditions, and uses the image as a target image to be processed by the image processing device 130 .

[0043] When the particle group is, for example, a sintered body group, in the object image generation process 120, the sintered body group is embedded in a resin to form a block, and a cross section obtained by cutting the formed block is photographed, thereby generating an object image in which a non-homogeneous target group is densely packed.

[0044] Furthermore, when the particle is, for example, a spheroid, in the target image generation process 120 , a culture solution containing the three-dimensionally cultured spheroid is imaged using a microscope, thereby generating a target image in which a non-homogeneous target group is densely packed.

[0045] It should be noted that the target image generated by the target image generation process 120 is notified to the image processing device 130 .

[0046] An image processing program is installed in the image processing device 130 , and by executing the image processing program, the image processing device 130 functions as a region segmentation unit 131 and a statistical processing unit 132 .

[0047] The region segmentation unit 131 performs region segmentation on the object image including the target group, and notifies the statistical processing unit 132 of the object image obtained by segmentation for each target region. It should be noted that, in the present embodiment, the region segmentation unit 131 performs region segmentation by combining a method using image analysis with a method using deep learning. Specifically, the watershed method as an example of image analysis is combined with U-Net as an example of deep learning, and after using U-Net to generate images for the watershed method (background region image, foreground region image, boundary region image), the region segmentation is performed by the watershed method.

[0048] Thus, according to the region segmentation unit 131 of the present embodiment, even when an image in which a non-homogeneous group of objects is densely packed is used as a target image, highly accurate region segmentation can be achieved (the details will be described later).

[0049] The statistical processing unit 132 analyzes the target image segmented for each target region and performs statistical processing (for example, processing to total the number of targets, the size of the targets, etc.).

[0050] Furthermore, the statistical processing unit 132 changes the process conditions in the target process 110 based on the statistical processing results.

[0051] <Hardware Configuration of Image Processing Device>

[0052] Next, the hardware configuration of the image processing device 130 will be described. Figure 2 is a diagram showing an example of the hardware configuration of the image processing apparatus.

[0053] like Figure 2 As shown, the image processing device 130 includes a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a driver device 206. It should be noted that the hardware components of the image processing device 130 are connected to each other via a bus 207.

[0054] The processor 201 includes various computing devices such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit). The processor 201 reads various programs (eg, image processing programs) to the memory 202 and executes them.

[0055] The memory 202 includes a main storage device such as a ROM (Read Only Memory) or a RAM (Random Access Memory). The processor 201 and the memory 202 form a so-called computer, and the processor 201 executes various programs read from the memory 202 to make the computer realize various functions.

[0056] The auxiliary storage device 203 stores various programs and various data used when the processor 201 executes the various programs.

[0057] The I / F device 204 is a connection device that connects the operating device 210 and the display device 220 to the image processing device 130. It should be noted that the operating device 210 is an operating device for allowing the operator to input various instructions to the image processing device 130, and the display device 220 is a display device for providing a display screen to the operator. The communication device 205 is a communication device for communicating with an external device (not shown) via a network.

[0058] The drive device 206 is a device for setting the recording medium 230. The recording medium 230 mentioned here includes a medium that records information optically, electrically or magnetically, such as a CD-ROM, a floppy disk, a magneto-optical disk, etc. In addition, the recording medium 230 may include a semiconductor memory that records information electrically, such as a ROM, a flash memory, etc.

[0059] It should be noted that the various programs installed in the auxiliary storage device 203 can be installed by, for example, placing the distributed recording medium 230 in the drive device 206 and having the drive device 206 read out the various programs recorded in the recording medium 230. Alternatively, the various programs installed in the auxiliary storage device 203 can be installed by downloading from the network via the communication device 205.

[0060] <Specific example of process control system>

[0061] Next, a specific example of the process control system 100 when the target process is a sintering process will be described. Figure 3 2 is a diagram showing a specific example of a process control system. Hereinafter, a specific example in this embodiment will be described with respect to a case where the target process is a sintering process.

[0062] like Figure 3 As shown, after the sintering process 310 is performed and the sintered body group 340 is generated, the object image generation process 320 is performed.

[0063] like Figure 3As shown, when the target process is a sintering process, in the target image generating process 320, a sintered body group collecting process, a resin embedding process, a cutting process, a dyeing process, and a photographing process are executed.

[0064] In the sintered body group collecting process, the generated sintered body group 340 is collected. In the resin embedding process, the collected sintered body group 340 is embedded in a transparent resin to generate a block 321.

[0065] In the cutting process, the block 321 is cut at a plurality of locations to generate a plurality of blocks 322. In the staining process, the cut surfaces of the plurality of blocks 322 are stained, and in the imaging process, the stained cut surfaces are imaged to generate the object image 323.

[0066] In this way, when the target process is a sintering process, by photographing the cut surface of the sintered body embedded in the resin in a dyed and exposed state, a target image including a target group consisting of cut surfaces of a plurality of sintered bodies is generated.

[0067] <Details of the target image>

[0068] Next, the details of the object image 323 generated by the object image generation process 320 will be described. Figure 4 A diagram for explaining the details of a target image.

[0069] like Figure 4 As shown, the object image 323 has the following features.

[0070] ·The target group consisting of the cross sections of a plurality of sintered bodies is densely packed.

[0071] The target group includes targets of various shapes and sizes (eg, target areas 401 to 405 , etc.).

[0072] The inside of each target has uneven color due to various irregularities and shades of the cut surface of the sintered body (for example, target regions 401 to 405 ).

[0073] In the background region (such as reference numerals 411 to 413 ) which is a region other than the target, the color is uneven due to the presence or absence of a sintered body behind it.

[0074] <Functional Structure of Region Division Unit>

[0075] Next, the functional structure of the region segmentation unit 131 that segments the object image 323 into each target region is described. As described above, the region segmentation unit 131 combines the watershed method and U-Net, generates an image for the watershed method using U-Net, and then performs region segmentation using the watershed method. Therefore, in the following description of the functional structure of the region segmentation unit 131, first,

[0076] · Functional structure of the region segmentation unit when using the general watershed method

[0077] Explain, then

[0078] The functional configuration of the region segmentation unit 131 according to the present embodiment, which is a combination of the watershed method and the U-Net, will be described to clarify the features of the region segmentation unit 131 according to the present embodiment.

[0079] (1) Functional structure of the region segmentation unit when using the general watershed method

[0080] First, refer to Figure 6 and Figure 7 ,use Figure 5 The functional configuration of a region segmentation unit when a general watershed method is used will be described. Figure 5 1 is a diagram showing an example of the functional configuration of a region segmentation unit when a general watershed method is used. Figure 6 , Figure 7 FIG. 1 and FIG. 2 are specific examples of processing performed by the area segmentation unit when a general watershed method is used.

[0081] like Figure 5 As shown, the region segmentation unit 500 using the general watershed method includes a threshold processing unit 501 , a morphological transformation processing unit 502 , a distance transformation processing unit 503 , a difference processing unit 504 , a labeling processing unit 505 , and a watershed method processing unit 506 .

[0082] The threshold processing unit 501 processes the target image (for example, the target image 323 (see Figure 3 , Figure 4 , Figure 6 )) performs threshold processing, transforms pixels less than the threshold parameter into black, and transforms pixels above the threshold parameter into white, thereby generating a binary image. The binary image generated by the threshold processing unit 501 is notified to the morphological transformation processing unit 502.

[0083] The morphological transformation processing unit 502 generates a background region image (for example, a background region image 511 (see FIG. 5 )) by performing morphological transformation processing on the binary image generated by the threshold processing unit 501 based on the adjusted transformation parameters. Figure 6 )) and determine the background pixels. The background region image (e.g., background region image 511 (see Figure 6 )) notifies the distance transformation processing unit 503 and the difference processing unit 504.

[0084] The distance transformation processing unit 503 transforms the pixel value of each pixel outside the background area into a pixel value corresponding to the distance from the background pixel based on the adjusted transformation parameter in the background area image generated by the morphological transformation processing unit 502. Thus, the distance transformation processing unit 503 generates a distance image (e.g., a distance image 512 (see Figure 6 )).

[0085] In addition, the distance transformation processing unit 503 generates a distance image (for example, the distance image 512 (refer to Figure 6 )), pixels above the adjusted threshold parameter are determined as foreground pixels. In addition, the distance transformation processing unit 503 generates a foreground area image (for example, a foreground area image 513 (see Figure 6 )). The foreground area image generated by the distance transform processing unit 503 (for example, the foreground area image 513 (refer to Figure 6 ))Notify the difference processing unit 504.

[0086] The difference processing unit 504 calculates the background region image (for example, the background region image 511 (see Figure 6 )) and the foreground area image generated by the distance transform processing unit 503 (for example, the foreground area image 513 (refer to Figure 6 )). Thus, the difference processing unit 504 generates a boundary region image (for example, a boundary region image 514 (see Figure 7 )) and determine the boundary pixels.

[0087] The labeling processing unit 505 performs

[0088] Based on the background area image (for example, the background area image 511 (refer to Figure 6 ))Determine the background pixels,

[0089] Based on the foreground area image (for example, the foreground area image 513 (refer to Figure 6 ))Determine the foreground pixels,

[0090] Based on the boundary region image (e.g., boundary region image 514 (see Figure 7 )) respectively mark the boundary pixels determined by the method, thereby generating a marked image (for example, marked image 521 (refer to Figure 7)). The labeled image (for example, the labeled image 521 (see Figure 7 ))Notify the watershed processing unit 506.

[0091] The watershed processing unit 506 obtains a target image (for example, the target image 323 (see Figure 3 , Figure 4 , Figure 6 )) and a marked image (e.g., marked image 521 (refer to Figure 7 )), the object image is segmented into each target region by the watershed method. Thus, the watershed method processing unit 506 generates a region segmentation image (for example, a region segmentation image 522 obtained by color-separating in a manner that enables the regions of each target to be distinguished (see Figure 7 Alternatively, the watershed processing unit 506 generates a region segmentation image (for example, a region segmentation image 523 obtained by color-differentiating the contours of each target (see Figure 7 )) and output.

[0092] Here, in the case of the area segmentation process performed by the area segmentation unit 500 using the general watershed method, there are the following problems.

[0093] As described above, it is necessary to appropriately adjust the threshold parameters or transformation parameters in the threshold processing unit 501, the morphological transformation processing unit 502, and the distance transformation processing unit 503 of the region segmentation unit 500. On the other hand, in the case of the object image 323 (an image in which a non-homogeneous group of objects is densely packed), it is not easy to find a threshold parameter or transformation parameter that is suitable for any of the multiple objects.

[0094] So, for example, Figure 7 As shown in the target areas 402_1 and 402_2 in FIG. 1 , it is possible that the target area 402 that should have been divided into one target area 402 (see Figure 4 ) is divided into two target areas.

[0095] Or, for example, Figure 7 As shown in the target region 403 ′ in FIG. 8 , there is a possibility that the target region that should be divided into three target regions 403 to 405 is divided into one target region.

[0096] In other words, when an image containing a dense group of inhomogeneous objects is used as a target image, it is difficult to achieve high-precision region segmentation using a general watershed method.

[0097] (2) Functional structure of the area division unit

[0098] Next, refer to Fig. 9 , Fig.10 ,use Figure 8 The functional configuration of the region segmentation unit 131 according to the present embodiment, which combines the watershed method and U-Net, will be described. Figure 8 : is a diagram showing an example of the functional structure of the area segmentation unit. Fig. 9 , Fig.10 FIG. 1 and FIG. 2 are specific examples of processing performed by the area segmentation unit.

[0099] like Figure 8 As shown, the region segmentation unit 131 includes a U-Net mask processing unit 801 , a U-Net contour prediction unit 802 , an expanded contour image generation unit 803 , a difference processing unit 804 , a labeling processing unit 505 , and a watershed method processing unit 506 .

[0100] The U-Net mask processing unit 801 is an example of a binary image generating unit. The U-Net mask processing unit 801 is a deep learning model (an example of a first learning completed model) that has been learned to perform mask processing (predicting the target and background to generate a binary image) on an input image, and has a deep learning model for an object image. Specifically, the network structure of the U-Net mask processing unit 801 is U-Net, and in this embodiment, it has a method including using

[0101] Input image: an image including a target group consisting of a plurality of cross sections of sintered bodies embedded in a resin by photographing the cross section of the sintered body in a dyed and exposed state;

[0102] True value image: A deep learning model (i.e., a learned U-Net) that has been transferred from the learning data of an image in which the target group is processed as white and the background is processed as black in the input image. It should be noted that the learned deep learning model can be generated by performing transfer learning on an existing learned deep learning model. For example,

[0103] Input image: a photo of the vehicle.

[0104] True value image: An existing deep learning model that has been learned using an image in which the vehicle is set to white and the background is set to black as learning data is further transferred to the target image, thereby generating a deep learning model that has been transferred to the target image (i.e., a U-Net that has been transferred). By using the existing deep learning model for transfer learning, the amount of new learning data required can be reduced, and the time spent on learning can be reduced.

[0105] By inputting the object image (for example, the object image 323 (refer to Figure 3 , Figure 4 , Fig. 9 )), the U-Net mask processing unit 801 generates a binary image (for example, a binary image 811 (refer to Fig. 9 )). Thus, the U-Net mask processing unit 801 determines the background pixels.

[0106] It should be noted that the binary image (for example, the binary image 811 (see Fig. 9 )) notifies the U-Net contour prediction unit 802 and the difference processing unit 804.

[0107] The U-Net contour prediction unit 802 is an example of a prediction unit. The U-Net contour prediction unit 802 has a deep learning model (an example of a second learning model) that has been learned to predict contours from an input binary image. Specifically, the network structure of the U-Net contour prediction unit 802 is U-Net, and in this embodiment, it has a method including using

[0108] Input image: The target group is processed into white and the background is processed into black.

[0109] True value image: A deep learning model (i.e., a learned U-Net) that has been trained using learning data of an image in which the outline of each target in the target group is processed in white and the targets and background other than the outline are processed in black.

[0110] By inputting a binary image (for example, a binary image 811 (refer to Figure 8 )), the U-Net contour prediction unit 802 predicts the contour and outputs a contour image. It should be noted that the contour image output by the U-Net contour prediction unit 802 is notified to the expansion contour image generation unit 803.

[0111] The dilated contour image generation unit 803 generates a dilated contour image (for example, a dilated contour image 812 (see FIG. 1 )) by dilating the contour in the contour image output by the U-Net contour prediction unit 802. Fig. 9 )). Thus, the expanded contour line image generation unit 803 specifies the boundary pixels.

[0112] It should be noted that the expanded contour line image (for example, the expanded contour line image 812 (see Fig. 9 ))Notify the difference processing unit 804.

[0113] The difference processing unit 804 is an example of a foreground area image generating unit. The difference processing unit 804 calculates the binary image (for example, the binary image 811 (refer to Fig. 9 )) and the expanded contour image generated by the expanded contour image generating unit 803 (for example, the expanded contour image 812 (see Fig. 9 Thus, the difference processing unit 804 generates a foreground area image (for example, the foreground area image 813 (refer to Fig. 9 )) and determine the foreground pixels.

[0114] The labeling processing unit 505 performs

[0115] Based on a binary image (for example, binary image 811 (refer to Fig. 9 ))Determine the background pixels,

[0116] Based on the dilated contour image (e.g., dilated contour image 812 (see Fig. 9 )) determined boundary pixels, based on the foreground area image (e.g., foreground area image 813 (refer to Fig. 9 )) respectively mark the foreground pixels determined by the method, thereby generating a marked image (for example, marked image 821 (refer to Fig.10 )). The labeled image (e.g., labeled image 821 (see Fig.10 ))Notify the watershed processing unit 506.

[0117] The watershed processing unit 506 obtains a target image (for example, the target image 323 (see Figure 3 , Figure 4 , Fig. 9 )) and a marked image (e.g., marked image 821 (refer to Fig.10 )), the target image is segmented into each target region by the watershed method. Thus, the watershed method processing unit 506 generates a region segmentation image (for example, a region segmentation image 822 obtained by color-separating in a manner that enables the regions of each target to be distinguished (see Fig.10 Alternatively, the watershed processing unit 506 generates a region segmentation image (for example, a region segmentation image 823 obtained by color-differentiating the contours of each target (see Fig.10 )) and output.

[0118] Here, in the case of the region segmentation process performed by the region segmentation unit 131 according to the present embodiment that combines the watershed method and U-Net, there are the following advantages.

[0119] In the region segmentation process performed by the region segmentation unit 500, various parameters need to be adjusted when generating a background region image, a foreground region image, and a boundary region image for determining background pixels, foreground pixels, and boundary pixels. In contrast, in the case of the region segmentation process performed by the region segmentation unit 131 according to the present embodiment, since a learned deep learning model is used, it is not necessary to adjust various parameters.

[0120] When outputting a contour image for determining boundary pixels, the target region and the background region are predicted from the object image to generate a binary image, and then the contour is predicted and the contour image is output, instead of directly predicting the contour from the object image. Thus, in the case of the region segmentation unit 131 according to the present embodiment, the contour can be predicted with high accuracy. This is because, in the case of a binary image, since the uneven color inside the object is deleted, the contour can be predicted in a manner that is not affected by the uneven color inside the object, unlike the case of directly predicting the contour from the object image. In addition, when the target region and the background region are predicted from the object image and a binary image is generated, the influence of the uneven color inside the object can be reduced compared to the case of predicting the contour from the object image. That is, by combining two deep learning models that have been learned, the contour is predicted in two stages, so that the influence of a non-homogeneous target group can be reduced, and the contour can be predicted with high accuracy using less learning data.

[0121] Therefore, the region segmentation unit 131 according to the present embodiment can realize highly accurate region segmentation even when an image in which a non-homogeneous group of objects is densely packed is used as a target image.

[0122] <Functional Structure of Statistical Processing Unit>

[0123] Next, the functional configuration of the statistical processing unit 132 will be described. Fig.11 FIG. 1 is a diagram showing an example of the functional configuration of the statistical processing unit. Fig.11 As shown, the statistical processing unit 132 includes a target extraction unit 1101 , an analysis unit 1102 , a totalization unit 1103 , and a condition change unit 1104 .

[0124] The object extraction unit 1101 obtains a region segmentation image (for example, the region segmentation image 822 (see Fig.10 ) or the region segmentation image 823 (refer to Fig.10 , Fig.11 )). In addition, the object extraction unit 1101 extracts the object based on the acquired region segmentation image (for example, the region segmentation image 822 (see Fig.10 ) or the region segmentation image 823 (refer to Fig.10 , Fig.11)), generate an image 1121 etc. in units of targets.

[0125] The analyzing unit 1102 analyzes the image 1121 or the like in units of objects generated by the object extracting unit 1101 , and identifies, for example, the type and size of the object.

[0126] The totaling unit 1103 totals the types, sizes, etc. of the targets specified by the analyzing unit 1102 and outputs the totaling result.

[0127] The condition changing unit 1104 changes the process conditions of the sintered body based on the summing result outputted from the summing unit 1103 , and outputs the changed process conditions.

[0128] <Process Control Process>

[0129] Next, the flow of the process control processing performed by the process control system 100 will be described. Fig.12 : is a flowchart showing the flow of the process control processing.

[0130] In step S1201 , a sintered body group 340 is generated by performing the sintering process 310 .

[0131] In step S1202 , an object image 323 including a heterogeneous target group is generated by executing the object image generation process 320 .

[0132] In step S1203, the image processing device 130 performs a region segmentation process for segmenting the target image 323 into each target region. It should be noted that the details of the region segmentation process will be described later.

[0133] In step S1204 , the image processing device 130 analyzes the region segmentation image 822 or the region segmentation image 823 generated by performing the region segmentation process, and outputs a summary result by performing statistical processing.

[0134] In step S1205 , the image processing device 130 changes the process conditions based on the totaling result, and outputs the changed process conditions.

[0135] In step S1206 , the sintering process 310 is performed based on the changed process conditions, thereby generating a sintered body group 340 .

[0136] <Region Segmentation Process>

[0137] Next, the details of the region segmentation process (step S1203) will be described. Fig.13 : is a flowchart showing the flow of the area segmentation process.

[0138] In step S1301 , the image processing device 130 acquires the target image 323 .

[0139] In step S1302 , the image processing device 130 generates a binary image 811 from the object image 323 using a deep learning model whose network structure is U-Net and has been learned, and determines background pixels.

[0140] In step S1303 , the image processing device 130 uses a deep learning model whose network structure is U-Net, predicts contours from the binary image, and generates a contour image.

[0141] In step S1304 , the image processing device 130 dilates the contour line image to generate a dilated contour line image 812 , thereby determining the boundary pixels.

[0142] In step S1305 , the image processing device 130 generates a foreground area image 813 by the difference between the binary image 811 and the dilated contour line image 812 , and determines the foreground pixels.

[0143] In step S1306 , the image processing device 130 uses the determined background pixels, boundary pixels, and foreground pixels to segment the object image 323 into each target area using the watershed method.

[0144] <Conclusion>

[0145] As is clear from the above description, the image processing device 130 according to the first embodiment performs the following processing.

[0146] Generate a binary image from the object image using a deep learning model (network structure = U-Net) that binarizes the object image into a target area and a background area and generates a binary image.

[0147] Using a deep learning model (network structure = U-Net) that predicts the contour of a target area in a binary image, the contour of the target area is predicted in a binary image generated from the object image.

[0148] Dilation is performed on the predicted contour to generate a dilated contour image.

[0149] Generate a foreground area image based on the difference between the binary image generated from the object image and the dilated contour image.

[0150] The object image is segmented into each target area by a watershed method using background pixels determined based on a binary image generated from the object image, boundary pixels determined based on a dilated contour image, and foreground pixels determined based on a foreground area image.

[0151] Thus, according to the first embodiment, when generating a background area image, a foreground area image, and a boundary area image for determining background pixels, foreground pixels, and boundary pixels, it is not necessary to adjust various parameters. In addition, according to the first embodiment, compared with the case where the contour line is directly predicted from the object image, the contour line can be predicted with high accuracy using less learning data.

[0152] Therefore, according to the first embodiment, it is possible to improve the segmentation accuracy when segmenting an image including a heterogeneous target group into regions.

[0153] [Second embodiment]

[0154] In the first embodiment described above, the case of using U-Net as the network structure of the deep learning model is described. However, the network structure of the deep learning model is not limited to U-Net. For example, as long as the network structure has a convolutional layer and can output an image by inputting an image, any other deep learning model can be applied. However, it is preferred that the deep learning model can output the contour of each target so that the contour line of the target area can be predicted more appropriately.

[0155] In the first embodiment, the case where the particle group is a sintered body group is described as a specific example. However, the particle group generated by the target process 110 is not limited to a sintered body group, but may be any other particle group. For example, any other particle group includes spherical bodies.

[0156] In addition, in the above-mentioned first embodiment, the case where the region segmentation unit 131 outputs the region segmentation image 822 or the region segmentation image 823 is described, but the image output by the region segmentation unit 131 is not limited to the region segmentation image 822 or the region segmentation image 823, and it may also be other images.

[0157] In the first embodiment, the statistical processing unit 132 outputs the total result obtained by totaling the types and sizes of each object. However, the items output as the total result are not limited to these.

[0158] In the first embodiment, the process conditions are changed based on the summary results and the changed process conditions are output. However, the items output by the statistical processing unit 132 to the target process 110 are not limited to this.

[0159] It should be noted that the present invention is not limited to the schemes listed in the above embodiments and the combinations of other elements shown here. These aspects can be changed within the scope of the present invention and can be appropriately determined according to the application form.

[0160] This application claims priority based on Japanese patent application No. 2022-091556 filed on June 6, 2022, and the entire contents of the Japanese patent application are cited by reference in this application.

[0161] Explanation of symbols

[0162] 100: Process control system

[0163] 130: Image processing device

[0164] 131: Area division

[0165] 132: Statistical Processing Department

[0166] 323: Object Image

[0167] 505: Annotation Processing Unit

[0168] 506: Watershed Processing Unit

[0169] 801: U-Net mask processing unit

[0170] 802: U-Net contour prediction unit

[0171] 803: Expanded contour image generation unit

[0172] 804: Difference processing unit.

Claims

1. An image processing device, comprising: A binary image generating unit generates a binary image from the object image using a first learned model that binarizes the object image into a target region and a background region and generates a binary image; a prediction unit that predicts a contour line of a target area in a binary image generated from the object image using a second learned model that predicts a contour line of a target area in a binary image; an expanded contour line image generating unit, which performs expansion processing on the predicted contour line to generate an expanded contour line image; a foreground area image generating unit that generates a foreground area image based on the binary image generated from the object image and the dilated contour line image; as well as The region segmentation unit segments the object image into each target region by a watershed method using background pixels determined based on a binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground region image.

2. The image processing device according to claim 1, wherein: The binary image generating unit generates the binary image from the target image using the learned U-Net as the first learned model.

3. The image processing device according to claim 2, wherein: The prediction unit generates a contour line image from a binary image generated from the target image using the learned U-Net as the second learned model.

4. The image processing device according to claim 3, wherein: The foreground area image generating unit generates the foreground area image by calculating a difference between a binary image generated from the object image and the dilated contour line image.

5. The image processing device according to any one of claims 1 to 4, wherein: The target image is an image obtained by imaging a particle group under predetermined conditions.

6. The image processing device according to claim 5, comprising: The statistical processing unit analyzes and performs statistical processing on images obtained by dividing the target image into each target area.

7. The image processing device according to claim 6, wherein: The statistical processing unit changes process conditions of a process for generating the particle group based on a result of the statistical processing.

8. An image processing method, wherein a computer executes the following steps: A step of generating a binary image from the object image using a first learning completion model that binarizes the object image into a target region and a background region and generates a binary image; A step of predicting a contour line of a target area in a binary image generated from the object image using a second learned model for predicting a contour line of a target area in a binary image; A step of dilating the predicted contour line to generate a dilated contour line image; A step of generating a foreground area image based on the binary image generated from the object image and the dilated contour image; as well as A step of segmenting the object image into each target area by a watershed method using background pixels determined based on a binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground area image.

9. An image processing program for causing a computer to execute the following steps: A step of generating a binary image from the object image using a first learning completion model that binarizes the object image into a target region and a background region and generates a binary image; A step of predicting a contour line of a target area in a binary image generated from the object image using a second learned model for predicting a contour line of a target area in a binary image; A step of dilating the predicted contour line to generate a dilated contour line image; A step of generating a foreground area image based on the binary image generated from the object image and the dilated contour image; as well as A step of segmenting the object image into each target area by a watershed method using background pixels determined based on a binary image generated from the object image, boundary pixels determined based on the dilated contour image, and foreground pixels determined based on the foreground area image.

Images