Image segmentation program, image segmentation device, and model generation method

Abstract

To divide properly an image containing multiple regions.SOLUTION: A computer inputs a received image into a first model. The first model is generated by machine learning based on training data including a composite image obtained by combining a plurality of images each containing one region and information indicating the state of combination of the composite images. The computer inputs to a second model a first image among a plurality of images output by the first model in response to the input of the images, the plurality of images being obtained by dividing the image. The second model is generated by machine learning based on training data including images containing one region and images containing multiple regions. The computer determines whether to input the first image to the first model based on the result output by the second model in response to the input of the first image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to image segmentation technology. [Background technology]

[0002] Electronic and paper documents often contain multiple images that are combined according to a certain pattern. When analyzing a composite image, it is desirable to split the composite image into multiple images and process each image separately.

[0003] Document layout analysis in OCR (Optical Character Recognition) often uses rule-based segmentation methods that do not use training data. These segmentation methods use, for example, human-defined rules and human-created parameters or filters to segment a document into multiple regions. Other methods that may be used include connected component analysis (CCA), run-length smoothing algorithm (RLSA), clustering, and edge detection.

[0004] On the other hand, segmentation methods using training data are often applied to semantic segmentation using machine learning, in which segmentation and labeling are performed using a single model, and labels can be assigned to any shape or polygon.

[0005] DeepLab is an example of semantic segmentation (see, for example, Non-Patent Document 1). Semantic segmentation using synthetic data is also known (see, for example, Non-Patent Document 2). The largest dataset for document layout analysis is also known (see, for example, Non-Patent Document 3).

[0006] Page layout analysis using semantic segmentation is also known (see, for example, Non-Patent Document 4). Techniques for segmenting manga images and assigning indexes are also known (see, for example, Non-Patent Document 5 and Non-Patent Document 6).

[0007] There is also known a document processing device that effectively uses divided object data included in a page such as a double-page spread in a document as one object data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2008-22159 [Non-patent literature]

[0009] [Non-Patent Document 1] Liang-Chieh Chen et al., "Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation", arXiv:1802.02611, Cornell University, 2018. [Non-patent document 2] K. Srivastava et al., "Multi Modal Semantic Segmentation using Synthetic Data", arXiv:1910.13676, Cornell University, 2019. [Non-patent document 3] X. Zhong et al., "PubLayNet: largest dataset ever for document layout analysis", arXiv:1908.07836, Cornell University, 2019. [Non-patent document 4] Y. Li et al., "DeepLayout: A Semantic Segmentation Approach to Page Layout Analysis", In: Huang et al. (Eds) Intelligent Computing Methodologies, ICIC 2018, Lecture Notes in Computer Science, volume 10956, pages 266-277, 2018 [Non-patent document 5] C. Rigaud, "Segmentation and indexation of complex objects in comic book images", Electronic Letters on Computer Vision and Image Analysis 14(3), 2014. [Non-patent document 6] N. Nguyen et al., "Digital Comics Image Indexing Based on Deep Learning", Journal of Imaging, 4, 89, 2018. Summary of the Invention [Problem to be solved by the invention]

[0010] The size of the regions into which a composite image contained in a document should be divided varies depending on the type of document and the purpose of document analysis, making it difficult to determine the appropriate size of the regions after division.

[0011] This problem is not limited to the case of dividing a composite image contained in a document, but occurs when dividing various images.

[0012] In one aspect, the present invention aims to appropriately segment an image including multiple regions. [Means for solving the problem]

[0013] In one example, the image segmentation program causes a computer to perform the following processes.

[0014] The computer inputs the received image into a first model, which is generated by machine learning based on training data including a composite image formed by combining multiple images, each of which includes a single region, and information indicating the state of the composite image combination.

[0015] The computer inputs a first image from among the multiple images output by the first model in response to an input image and obtained by dividing the image into multiple images, into a second model. The second model is generated by machine learning based on training data including an image containing one region and an image containing multiple regions.

[0016] The computer determines whether or not to input the first image to the first model based on the result output by the second model in response to the input of the first image. [Effects of the Invention]

[0017] According to one aspect, an image including a plurality of regions can be appropriately divided. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is a functional configuration diagram of an image segmentation device. [Figure 2] 10 is a flowchart of an image division process. [Figure 3] FIG. 1 is a functional configuration diagram showing a specific example of an image segmentation device. [Figure 4] FIG. 10 is a diagram showing a horizontal pattern. [Figure 5] FIG. 10 shows a vertical pattern. [Figure 6] FIG. 10 is a diagram showing a grid pattern. [Figure 7] FIG. 1 illustrates a recursive pattern. [Figure 8] FIG. 1 illustrates a training process. [Figure 9] FIG. 10 is a diagram illustrating a first image division process. [Figure 10] FIG. 10 is a diagram illustrating a second image division process. [Figure 11] 10 is a flowchart of a training process. [Figure 12] 10 is a flowchart showing a specific example of image division processing. [Figure 13] FIG. 2 is a hardware configuration diagram of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0020] 1 shows an example of the functional configuration of an image segmentation device according to an embodiment. The image segmentation device 101 in FIG.

[0021] Fig. 2 is a flowchart showing an example of image segmentation processing performed by the control unit 111 in Fig. 1. First, the control unit 111 inputs a received image to a first model (step 201). The first model is generated by machine learning based on training data including a composite image obtained by combining multiple images, each of which includes one region, and information indicating the combined state of the composite image.

[0022] Next, the control unit 111 inputs a first image among the multiple images output by the first model in response to the input image and obtained by dividing the image into a second model (step 202). The second model is generated by machine learning based on training data including an image including one region and an image including multiple regions.

[0023] Next, the control unit 111 determines whether or not to input the first image to the first model based on the result output by the second model in response to the input of the first image (step 203).

[0024] According to the image segmentation device 101 in FIG. 1, an image including a plurality of regions can be appropriately segmented.

[0025] Fig. 3 shows a specific example of the image segmentation device 101 in Fig. 1. The image segmentation device 301 in Fig. 3 includes a first generation unit 311, a second generation unit 312, a third generation unit 313, a control unit 314, an output unit 315, and a storage unit 316, and operates in either a machine learning mode or an operational mode. The control unit 314 corresponds to the control unit 111 in Fig. 1.

[0026] In the machine learning mode, the storage unit 316 stores a first unit image set 321, a first composite image set 322, an image set 323, and a combination pattern 324. The combination pattern 324 is an example of a predetermined combination pattern.

[0027] The first unit image set 321, the first composite image set 322, the image set 323, and the combination pattern 324 are selected according to the application of the image segmentation device 301. Applications of the image segmentation device 301 include segmenting images contained in documents such as academic papers, segmenting frames of comic images, etc.

[0028] The first unit image set 321 includes a plurality of unit images that have been manually labeled by a human. Each unit image is an image that includes only one region, and a label indicating that it is a unit image is attached to each unit image. The first composite image set 322 includes a plurality of composite images that have been manually labeled by a human. Each composite image is an image that includes a plurality of unit images, and a label indicating that it is a composite image is attached to each composite image.

[0029] The image set 323 includes a plurality of unlabeled images. Each image is either a unit image or a composite image. The number of images included in the image set 323 is greater than the sum of the number of unit images included in the first unit image set 321 and the number of composite images included in the first composite image set 322. The combination pattern 324 represents rules that define the combination state of the plurality of unit images included in various composite images.

[0030] 4 to 7 show examples of combination patterns 324 for combining rectangular unit images. Fig. 4 shows an example of a horizontal pattern in which unit image A and unit image B are combined by arranging them in the horizontal direction.

[0031] Figure 4(a) shows the H_fit <r>An example of (A,B) is shown. H_fit <r>In (A,B), a composite image is generated by stacking unit images A and B horizontally so that the vertical length (height) of unit image A matches the height of unit image B. r represents the ratio of the width of unit image A to the horizontal length (width) of the composite image. In this example, r=0.3. The aspect ratio of at least one of unit images A and B is changed.

[0032] Figure 4(b) shows the H_up <r>An example of (A,B) is shown. <r>In (A,B), a composite image is generated by stacking unit images A and B horizontally so that the top edges of unit images A and B are at the same height. r represents the ratio of the width of unit image A to the width of the composite image. In this example, r=0.66. The aspect ratios of unit images A and B remain unchanged.

[0033] Figure 4(c) shows the H_center <r>An example of (A,B) is shown. H_center <r>In (A,B), a composite image is generated by stacking unit images A and B horizontally so that the centers of gravity of unit images A and B are at the same height. r represents the ratio of the width of unit image A to the width of the composite image. In this example, r=0.66. The aspect ratios of unit images A and B are not changed.

[0034] Figure 4(d) shows H_down <r>An example of (A,B) is shown. H_down <r>In (A,B), a composite image is generated by stacking unit images A and B horizontally so that the bottom edges of unit images A and B are at the same height. r represents the ratio of the width of unit image A to the width of the composite image. In this example, r=0.66. The aspect ratios of unit images A and B remain unchanged.

[0035] FIG. 5 shows an example of a vertical pattern in which unit images A and B are combined in a vertical direction.

[0036] Figure 5(a) shows V_left <r>An example of (A,B) is shown. V_left <r>In (A,B), a composite image is generated by stacking unit images A and B vertically so that the left edge of unit image A and the left edge of unit image B are aligned horizontally. r represents the ratio of the height of unit image A to the height of the composite image. In this example, r=0.3. The aspect ratios of unit images A and B are not changed.

[0037] Figure 5(b) shows the V_right <r>An example of (A,B) is shown. V_right <r>In (A,B), a composite image is generated by stacking unit images A and B vertically so that the right edges of unit images A and B are aligned horizontally. r represents the ratio of the height of unit image A to the height of the composite image. In this example, r=0.3. The aspect ratios of unit images A and B are not changed.

[0038] Figure 5(c) shows the V_fit <r>An example of (A,B) is shown. <r>In (A,B), a composite image is generated by stacking unit images A and B vertically so that the widths of unit images A and B match. r represents the ratio of the height of unit image A to the height of the composite image. In this example, r=0.6. The aspect ratio of at least one of unit images A and B is changed.

[0039] Figure 6 shows the grid pattern GRID, which combines multiple unit images in a grid pattern.<col,row> Here is an example of (img[col*row]), where col represents the number of columns in the grid, and row represents the number of rows in the grid. GRID<col,row> In (img[col*row]), a composite image is generated by arranging col*row unit images in a grid pattern. In this example, col=2, row=3, and unit images A to F are arranged in a grid pattern of 3 rows and 2 columns.

[0040] Figure 7 shows the H_down <r>(A,B) and V_right <r>Recursive pattern H_down, which is a recursive combination of (A,B) <r>(A,V_right (B,C)) is shown. First, V_right According to (B,C), image B and image C are stacked vertically so that the right edge of image B and the right edge of image C are aligned at the same horizontal position. In this example, p=0.3.

[0041] Next, H_down <r> (A,V_right< / r> According to (B, C), image A and the combined image of image B and image C are stacked horizontally so that the bottom ends of image A and image C are aligned at the same height. In this example, r=0.6.

[0042] 8 shows an example of training processing performed in machine learning mode. The first generation unit 311 provides the first unit image set 321 and the first composite image set 322 as training data to the machine learning model, and performs supervised machine learning to generate a trained model, stopper 331. In supervised machine learning, labels assigned to each unit image and each composite image are used as training data.

[0043] In the operation mode, the stopper 331 outputs a determination result indicating whether the input image is a unit image or a composite image. The stopper 331 is an example of a second model.

[0044] Next, the second generation unit 312 inputs each image included in the image set 323 to the stopper 331, and classifies each image into either a unit image or a composite image based on the determination result output from the stopper 331. Then, the second generation unit 312 selects only the unit images from the image set 323 to generate a second unit image set 325, and stores it in the storage unit 316.

[0045] By using the stopper 331 to classify each image into either a unit image or a composite image, it is possible to extract a sufficient number of unit images from a large number of images and generate the second unit image set 325. This eliminates the need to extract unit images manually, and makes it possible to easily generate the second unit image set 325 that includes more unit images than the first unit image set 321.

[0046] Next, the second generation unit 312 generates a plurality of composite images by combining the plurality of images included in the second unit image set 325 based on the combination pattern 324. Then, the second generation unit 312 assigns identification information indicating the combination pattern 324 of each generated composite image as training data for that composite image, generates a second composite image set 326, and stores it in the storage unit 316. The identification information indicating the combination pattern 324 is an example of information indicating the combination state of the composite image.

[0047] By combining a plurality of images based on the combination pattern 324, a second composite image set 326 including a large number of composite images corresponding to various combination states can be easily generated.

[0048] Next, the second generation unit 312 provides the second composite image set 326 as training data to a machine learning model, and performs supervised machine learning to generate a trained model, a splitter 332. In the supervised machine learning, the identification information assigned to each composite image is used as training data.

[0049] In the operation mode, splitter 332 identifies the identification information of combination pattern 324 corresponding to the input image. Then, splitter 332 splits the input image into multiple images and outputs them according to combination pattern 324 indicated by the identified identification information. Splitter 332 is an example of a first model.

[0050] The second generation unit 312 may use the first set of unit images 321 instead of the second set of unit images 325 to generate a plurality of composite images.

[0051] Next, the third generation unit 313 combines the stopper 331 and the splitter 332 to generate an image division model 327 and stores it in the storage unit 316.

[0052] In the operational mode, the storage unit 316 stores a combination pattern 324, an image segmentation model 327, and a processing target image 328. The image 328 may be an image inserted into a document such as an academic paper, or a comic image including multiple frames.

[0053] First, the control unit 314 inputs the image 328 to the stopper 331, and the stopper 331 outputs a determination result in accordance with the input of the image 328. Then, the control unit 314 determines whether or not to input the image 328 to the splitter 332 based on the determination result output from the stopper 331.

[0054] If the determination result indicates a composite image, the control unit 314 accepts the image 328 and inputs it to the splitter 332, which then splits the image 328 into a plurality of images according to the combination pattern 324 and outputs the images.

[0055] Next, the control unit 314 recursively inputs each of the multiple images output from the splitter 332 to the stopper 331, and the stopper 331 outputs a determination result in accordance with the input of each image. Then, the control unit 314 determines whether or not to input each image to the splitter 332 based on the determination result output from the stopper 331.

[0056] If the determination result for any of the images indicates a composite image, the control unit 314 accepts the image and inputs it to the splitter 332, which then divides the accepted image into a plurality of images according to the combination pattern 324 and outputs them. By recursively repeating this image division using the stopper 331 and the splitter 332, an image that has been determined to be a composite image can be further divided into smaller images.

[0057] If the determination result for any image indicates a unit image, the control unit 314 determines not to input that image to the splitter 332 and cancels image division for that image. Then, the control unit 314 generates a division result 329 including the image for which image division has been canceled as a unit image, and stores it in the storage unit 316. The output unit 315 outputs the division result 329. This makes it possible to prevent an image having a determination result indicating a unit image from being divided into even smaller images.

[0058] If the determination result for image 328 indicates a unit image, control unit 314 stops image division for image 328, generates division result 329 including only image 328, and stores it in storage unit 316. Output unit 315 outputs division result 329.

[0059] The output unit 315 may output the segmentation result 329 to a post-processing step that performs image analysis. In this case, in the post-processing step, the multiple images included in the segmentation result 329 are analyzed individually or in association with each other.

[0060] 9 shows an example of the first image segmentation process. Image 901 in FIG. 9 is an image inserted into an academic paper document, and includes regions 911 to 913. Each region includes unit images of a graph showing experimental results. By inputting image 901 as image 328 to be processed to image segmentation device 301, image 901 is segmented into three unit images, regions 911 to 913.

[0061] 10 shows an example of the second image segmentation process. Image 1001 in FIG. 10 is an image inserted into an academic paper document, and includes regions 1011 to 1017. Each region includes unit images of a graph showing experimental results. By inputting image 1001 as image 328 to be processed into image segmentation device 301, image 1001 is segmented into seven unit images, regions 1011 to 1017.

[0062] 3, it is possible to easily determine whether or not to further divide each image by using the stopper 331 to determine whether or not each image obtained by dividing the image 328 to be processed is a unit image. As a result, it is possible to terminate image division by the splitter 332 when the image 328 has been divided into regions of an appropriate size.

[0063] Fig. 11 is a flowchart showing an example of a training process performed in the machine learning mode by the image segmentation device 301 in Fig. 3. First, the first generation unit 311 generates a stopper 331 by supervised machine learning using the first unit image set 321 and the first composite image set 322 (step 1101).

[0064] Next, the second generation unit 312 inputs each image included in the image set 323 to the stopper 331, and generates a second unit image set 325 by selecting images whose judgment results output from the stopper 331 indicate unit images (step 1102).

[0065] Next, the second generation unit 312 generates a plurality of composite images by combining the plurality of images included in the second unit image set 325 based on the combination pattern 324. Then, the second generation unit 312 assigns identification information indicating the combination pattern 324 of each composite image as training data for that composite image, thereby generating a second composite image set 326 (step 1103).

[0066] Next, the second generation unit 312 generates the splitter 332 by supervised machine learning using the second composite image set 326 (step 1104). Then, the third generation unit 313 combines the stopper 331 and the splitter 332 to generate the image segmentation model 327 (step 1105).

[0067] Fig. 12 is a flowchart showing a specific example of image segmentation processing performed in operation mode by the image segmentation device 301 of Fig. 3. First, the control unit 314 secures a stack area in the storage unit 316 and stores the image 328 to be processed in the stack area (step 1201). The stack area is a last-in-first-out (LIFO) storage area.

[0068] Next, the control unit 314 extracts the image 328 from the stack area and inputs it to the stopper 331, which then outputs the determination result for the image 328 (step 1202).The control unit 314 then checks whether the determination result indicates a unit image or a composite image (step 1203).

[0069] If the determination result indicates a unit image (1203, YES), the control unit 314 adds the image 328 to the segmentation result 329 (step 1205) and checks whether the stack area is empty (step 1206). In this case, since the stack area is empty (1206, YES), the output unit 315 outputs the segmentation result 329 that includes only the image 328 as a unit image (step 1207).

[0070] On the other hand, if the determination result indicates a composite image (1203, NO), the control unit 314 inputs the image 328 to the splitter 332 (step 1204). The splitter 332 splits the image 328 into a plurality of images according to the combination pattern 324 and outputs the split images. The control unit 314 then stores the plurality of images output from the splitter 332 in a stack area, and repeats the processing from step 1202 onwards for each image in the stack area.

[0071] If the determination result for the image extracted from the stack area indicates a unit image (1203, YES), the control unit 314 adds the image to the segmentation result 329 (step 1205) and checks whether the stack area is empty (step 1206). If the stack area is not empty (1206, NO), the control unit 314 repeats the processing from step 1202 onwards for the next image in the stack area.

[0072] If the determination result for the image extracted from the stack area indicates a composite image (1203, NO), the control unit 314 inputs the image to the splitter 332 (step 1204). The splitter 332 splits the input image into a plurality of images according to the combination pattern 324 and outputs the split images. The control unit 314 then stores the plurality of images output from the splitter 332 in the stack area, and repeats the processing from step 1202 onwards.

[0073] If the stack area becomes empty (1206, YES), the output unit 315 outputs the segmentation result 329 containing a plurality of images as unit images (step 1207).

[0074] The configurations of the image segmentation device 101 in Fig. 1 and the image segmentation device 301 in Fig. 3 are merely examples, and some of the components may be omitted or changed depending on the application or conditions of the image segmentation device. For example, in the image segmentation device 301 in Fig. 3, if the training process is executed by an external device, the first generation unit 311, the second generation unit 312, and the third generation unit 313 can be omitted.

[0075] 2, 11, and 12 are merely examples, and some processes may be omitted or changed depending on the configuration or conditions of the image segmentation device. For example, if the training process is performed by an external device, the training process in Fig. 11 can be omitted. In step 1103 of Fig. 11, if multiple composite images are generated using the first unit image set 321 instead of the second unit image set 325, the process of step 1102 can be omitted.

[0076] The combination patterns 324 shown in Figures 4 to 7 are merely examples, and the second generation unit 312 may generate multiple composite images using other combination patterns 324. The shape of the area of ​​the unit image may be a polygon other than a rectangle, for example. The training process shown in Figure 8 is merely an example, and some of the processes may be omitted or changed depending on the configuration or conditions of the image segmentation device. The image segmentation process shown in Figures 9 and 10 is merely an example, and the image segmentation form changes depending on the image 328 to be processed.

[0077] Fig. 13 shows an example of the hardware configuration of an information processing device (computer) used as the image segmentation device 101 in Fig. 1 and the image segmentation device 301 in Fig. 3. The information processing device in Fig. 13 includes a CPU (Central Processing Unit) 1301, a memory 1302, an input device 1303, an output device 1304, an auxiliary storage device 1305, a medium drive device 1306, and a network connection device 1307. These components are hardware and are connected to each other by a bus 1308.

[0078] The memory 1302 is, for example, a semiconductor memory such as a read-only memory (ROM) or a random access memory (RAM), and stores programs and data used in processing. The memory 1302 may operate as the storage unit 316 in FIG.

[0079] 1 by executing a program using the memory 1302. The CPU 1301 also operates as the first generation unit 311, the second generation unit 312, the third generation unit 313, and the control unit 314 in FIG.

[0080] The input device 1303 is, for example, a keyboard, a pointing device, etc., and is used to input instructions or information from a user or operator. The output device 1304 is, for example, a display device, a printer, etc., and is used to output inquiries or instructions to a user or operator and processing results. The processing results may be the segmentation result 329. The output device 1304 may operate as the output unit 315 in FIG. 3.

[0081] The auxiliary storage device 1305 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1305 may be a hard disk drive. The information processing device stores programs and data in the auxiliary storage device 1305 and can use them by loading them into the memory 1302. The auxiliary storage device 1305 may operate as the storage unit 316 in FIG. 3.

[0082] The medium drive device 1306 drives the portable recording medium 1309 and accesses the recorded contents thereof. The portable recording medium 1309 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, etc. The portable recording medium 1309 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, etc. A user or operator can store programs and data in the portable recording medium 1309 and load them into the memory 1302 for use.

[0083] In this way, the computer-readable recording medium that stores the program and data used in the processing is a physical (non-transitory) recording medium such as the memory 1302, the auxiliary storage device 1305, or the portable recording medium 1309.

[0084] The network connection device 1307 is a communication interface circuit connected to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network) and performs data conversion associated with communication. The information processing device receives programs and data from an external device via the network connection device 1307 and loads them into the memory 1302 for use. The network connection device 1307 may operate as the output unit 315 in FIG. 3.

[0085] 13, some components may be omitted depending on the purpose or conditions of the information processing device. For example, if an interface with a user or operator is not required, the input device 1303 and the output device 1304 may be omitted. If the portable recording medium 1309 or a communication network is not used, the medium drive device 1306 or the network connection device 1307 may be omitted.

[0086] Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims.

[0087] The following notes are further disclosed regarding the embodiment described with reference to FIGS. (Appendix 1) inputting the received image into a first model generated by machine learning based on training data including a composite image obtained by combining a plurality of images each including one region and information indicating a combination state of the composite image; inputting a first image among the plurality of images output by the first model in response to the input of the image and obtained by dividing the image into a second model generated by machine learning based on training data including an image including one region and an image including multiple regions; determining whether to input the first image to the first model based on a result output by the second model in response to the input of the first image; An image segmentation program that causes a computer to execute a process. (Appendix 2) 2. The image segmentation program of claim 1, further comprising causing the computer to execute a process of outputting the first image as an image including one region if it is determined that the first image should not be input to the first model. (Appendix 3) If it is determined that the first image is to be input to the first model, inputting the first image to the first model; inputting a second image of the plurality of images output by the first model in response to the input of the first image, the second image being obtained by dividing the first image, into the second model; 2. The image segmentation program according to claim 1, further causing the computer to execute a process. (Appendix 4) The image segmentation program according to any one of claims 1 to 3, characterized in that the composite image is generated by combining multiple images, each of which contains one area, based on a predetermined combination pattern. (Appendix 5) The image segmentation program according to any one of claims 1 to 4, wherein the plurality of images each containing one region are selected from the plurality of images included in the image set based on the results output by the second model in response to input of each of the plurality of images included in the image set. (Appendix 6) inputting the received image into a first model generated by machine learning based on training data including a composite image obtained by combining a plurality of images each including one region and information indicating a combination state of the composite image; inputting a first image among the plurality of images output by the first model in response to the input of the image and obtained by dividing the image into a second model generated by machine learning based on training data including an image including one region and an image including multiple regions; determining whether to input the first image to the first model based on a result output by the second model in response to the input of the first image; An image segmentation device comprising a control unit that executes processing. (Appendix 7) The image segmentation device described in Appendix 6, further comprising an output unit that outputs the first image as an image including one region when it is determined that the first image should not be input to the first model. (Appendix 8) The control unit If it is determined that the first image is to be input to the first model, inputting the first image to the first model; inputting a second image of the plurality of images output by the first model in response to the input of the first image, the second image being obtained by dividing the first image, into the second model; 7. The image segmentation device according to claim 6, further comprising: (Appendix 9) The image segmentation device described in any one of Appendices 6 to 8, characterized in that the composite image is generated by combining multiple images, each of which contains one area, based on a predetermined combination pattern. (Appendix 10) The image segmentation device according to any one of claims 6 to 9, wherein the plurality of images each including one region are selected from the plurality of images included in the image set based on the results output by the second model in response to input of each of the plurality of images included in the image set. (Appendix 11) a second model that outputs information for determining whether to input a first image into a first model in response to an input of the received image among the plurality of images output from the first model, by machine learning based on training data including an image including one region and an image including a plurality of regions; generating the first model by machine learning based on training data including a composite image obtained by combining a plurality of images each including one region and information indicating a combination state of the composite image; combining the first model and the second model to generate an image segmentation model. A model generation method characterized in that the processing is executed by a computer. (Appendix 12) 12. The model generation method of claim 11, wherein the computer further performs a process of generating the composite image by combining the plurality of images, each of which includes one region, based on a predetermined combination pattern. (Appendix 13) 13. The model generation method according to claim 11 or 12, wherein the computer further executes a process of selecting, from the images included in the image set, a plurality of images each including one region, based on the results output by the second model in response to input of each of the plurality of images included in the image set. [Explanation of symbols]

[0088] 101, 301 Image segmentation device 111, 314 Control section 311 1st generation part 312 Second generation part 313 Third generation part 315 Output section 316 Storage section 321 First Unit Image Set 322 First synthetic image set 323 Image Set 324 combination patterns 325 Second Unit Image Set 326 Second synthetic image set 327 Image Segmentation Model 328, 901, 1001 images 329 division results 331 Stopper 332 Splitter 911~913, 1011~1017 area 1301 CPU 1302 memory 1303 Input Device 1304 Output Device 1305 Auxiliary storage device 1306 Media drive unit 1307 Network connection device 1308 Bus 1309 Portable recording media < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r> < / r>

Claims

1. a first image is input to a second model that is generated by machine learning based on training data including a unit image including one region and a composite image generated by combining a plurality of unit images based on a predetermined combination pattern, and that outputs a determination result as to whether the input image is a unit image or a composite image; a first process that, when the determination result output by the second model in response to the input of the first image is determined to be a composite image, inputs the first image to a first model that divides an input image into a plurality of images and outputs the divided images; The image segmentation program according to claim 1, wherein the first model outputs an image obtained by dividing an input image into a plurality of regions.

2. a second process of inputting a plurality of images obtained by dividing the first image, which are output by the first model in response to the input of the first image, into the second model; 2. The image segmentation program according to claim 1, further comprising: a program for causing the computer to execute the program;

3. The first process and the second process are repeatedly performed recursively.

3. The image segmentation program according to claim 2.

4. 4. The image segmentation program according to claim 1, further comprising causing the computer to execute a process of outputting the first image as a unit image without inputting the first image to the first model when the judgment result output by the second model in response to the input of the first image is that the first image is a unit image.

5. the first model is a model generated by machine learning based on training data including the composite image and information indicating the combination pattern of the composite image, An image segmentation program according to any one of claims 1 to 4, characterized in that in the training process of the first model, a second composite image generated by combining multiple second unit images determined to be the unit images based on the judgment result of the second model based on the combination pattern is used as the training data for the first model.

6. a first image is input to a second model that is generated by machine learning based on training data including a unit image including one region and a composite image generated by combining a plurality of unit images based on a predetermined combination pattern, and that outputs a determination result as to whether the input image is a unit image or a composite image; a control unit that executes a first process of inputting the first image to a first model that divides an input image into a plurality of images and outputs the divided images when the determination result output by the second model in response to the input of the first image is determined to be a composite image; The image segmentation device according to claim 1, wherein the first model outputs an image obtained by dividing an input image into a plurality of regions.

7. generating a second model by machine learning based on training data including a unit image including one region and a composite image including a plurality of unit images; generating a first model by machine learning based on training data including the composite image and information indicating a combination pattern of the composite image; combining the first model and the second model to generate an image segmentation model; A model generation method in which processing is performed by a computer, the second model outputs a determination result as to whether an input image is a unit image or a composite image; the first model divides an input image into a plurality of images and outputs the images; inputting a first image into the second model, and when a determination result output by the second model in response to the input of the first image indicates that the image is a synthetic image, inputting the first image into the first model; A model generation method comprising:

Images