Device, method, and program

Abstract

To provide a technique for implementing efficient annotation operation.SOLUTION: A device includes: a region extraction processing unit which creates a second image by extracting one or more target regions from a first image, based on a model configured to receive, as input, an image including one or more targets and extract target regions representing the targets; and a UI unit which suppresses displaying the one or more target regions included in the second image, in displaying the second image on a screen for creating a third image for re-training the model by correcting extraction failure of the second image.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present disclosure relates to an apparatus, a method, and a program. [Background technology]

[0002] There is known a technique for determining the germination state of seeds in a photographed image using a classification model trained using machine learning technology (for example, Patent Document 1, etc.). To train such a classification model, it is necessary to prepare images in which the germination state of seed regions in the photographed image is labeled as training data.

[0003] The task of specifying a seed region in a captured image is called annotation, and an annotation tool (for example, Non-Patent Document 1) is generally used. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-157548 [Non-patent literature]

[0005] [Non-Patent Document 1] Visual Geometry Group - University of Oxford, Internet<URL:https: / / www.robots.ox.ac.uk / ~vgg / software / via / > Summary of the Invention [Problem to be solved by the invention]

[0006] However, in order to specify the seed region, it is necessary to manually plot pixels corresponding to the outline of the seed one by one in the captured image, which requires a lot of time and effort.

[0007] The present disclosure has been made in consideration of the above points, and provides a technology that can realize efficient annotation work. [Means for solving the problem]

[0008] An apparatus according to one aspect of the present disclosure includes a region extraction processing unit that creates a second image by extracting one or more target regions from a first image based on a model that extracts target regions representing the objects when an image containing the objects is input, and a UI unit that suppresses the display of one or more target regions included in the second image when displaying the second image on a screen that creates a third image for re-training the model by correcting extraction omissions in the second image. [Effects of the Invention]

[0009] A technology is provided that enables efficient annotation work. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the overall configuration of a data creation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a detailed functional configuration of an annotation processing unit according to the present embodiment. [Figure 3] 10 is a flowchart illustrating an example of annotation processing according to the present embodiment. [Figure 4] FIG. 10 is a diagram (part 1) showing an example of an annotation screen. [Figure 5] FIG. 10 is a diagram (part 2) showing an example of the annotation screen. [Figure 6] FIG. 10 is a diagram illustrating an example of an extracted region image. [Figure 7] FIG. 4 is a diagram illustrating an example of a detailed functional configuration of an area extraction processing unit according to the present embodiment. [Figure 8] 1 is a flowchart (Example 1) illustrating an example of a region extraction process according to the present embodiment. [Figure 9]10 is a flowchart showing an example of a region extraction process according to the present embodiment (Example 2); [Figure 10] 10 is a flowchart showing an example of a region extraction process according to the present embodiment (Example 3); [Figure 11] FIG. 10 is a diagram illustrating an example of a correction screen. [Figure 12] FIG. 2 is a diagram illustrating an example of a detailed functional configuration of a labeling processing unit according to the present embodiment. [Figure 13] 10 is a flowchart illustrating an example of a labeling process according to the present embodiment. [Figure 14] FIG. 4 is a diagram illustrating an example of a detailed functional configuration of a label change processing unit according to the embodiment. [Figure 15] 10 is a flowchart illustrating an example of a label change process according to the embodiment. [Figure 16] FIG. 10 is a diagram illustrating an example of a label change screen. DETAILED DESCRIPTION OF THE INVENTION

[0011] An embodiment of the present invention will be described below, which relates to a data creation system 1 that can create images in which labels indicating the germination state are assigned to seed regions in an image (hereinafter also referred to as labeled region extraction images), as learning data used to learn a classification model that determines the germination state of seeds in a captured image.

[0012] <Example of overall configuration of data creation system 1> An example of the overall configuration of a data creation system 1 according to this embodiment is shown in Fig. 1. As shown in Fig. 1, the data creation system 1 according to this embodiment includes a data creation device 10, one or more first user terminals 20, and one or more second user terminals 30. The data creation device 10, each of the first user terminals 20, and each of the second user terminals 30 are communicatively connected via a communication network 40 including, for example, the Internet.

[0013] The data creation device 10 creates a labeled region extraction image from a photographed image of one or more seeds placed in a petri dish, a growing tray, or the like. The data creation device 10 may be, for example, a personal computer (PC) or a general-purpose server. The photographed image is provided to the data creation device 10 by, for example, a seed company or the like.

[0014] The first user terminal 20 is a terminal of any type used by a person (hereinafter also referred to as a first user) who performs annotation on a captured image to extract a seed region and assign a label to the seed region. The first user terminal 20 may be, for example, a PC, a smartphone, a tablet terminal, a wearable device, or the like. Note that the first user may be, for example, a company employee who creates learning data.

[0015] The second user terminal 30 is a terminal used by a person (hereinafter also referred to as a second user) who checks the labels of the labeled region extraction image and changes them as necessary. The second user terminal 30 may be, for example, a PC, a smartphone, a tablet terminal, a wearable device, or the like. Note that the second user may be, for example, a person in charge of a seed company that provides photographed images.

[0016] Here, the data creation device 10 according to this embodiment includes an annotation processing unit 110, an area extraction processing unit 120, a labeling processing unit 130, and a label change processing unit 140. Each of these units is realized, for example, by a processor such as a CPU (Central Processing Unit) executing one or more programs installed in the data creation device 10. The data creation device 10 according to this embodiment also includes a storage unit 150. The storage unit 150 is realized, for example, by a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory.

[0017] The annotation processing unit 110 executes annotation processing to create a region extraction image (hereinafter also referred to as a first region extraction image) in which a first user performs annotation on a photographed image and extracts a seed region from the photographed image. Note that annotation means designating a seed region in a photographed image. Also, the region extraction image is data in which a group of coordinates (a set of x and y coordinates on an x ​​and y coordinate plane set on the photographed image) representing the outline of the seed region extracted from the photographed image is associated with the photographed image.

[0018] The region extraction processing unit 120 re-learns a given region extraction model using at least the first region extraction image, and then executes region extraction processing using this re-learned region extraction model to create a region extraction image (hereinafter also referred to as a second region extraction image) in which a seed region is extracted from a photographed image of a region extraction target (hereinafter also referred to as a target image). Here, the region extraction model is a machine learning model that takes an image as input and extracts an object region in the image. As the region extraction model, for example, an existing trained model such as a model that extracts seed regions related to seeds of different varieties from a photographed image is used.

[0019] The labeling processing unit 130 performs labeling processing to create a labeled extracted region image in which labels indicating the germination state are assigned to seed regions in the extracted region image.

[0020] The label change processing unit 140 executes label change processing to change the labels in the labeled region extraction image.

[0021] The storage unit 150 stores various data (for example, a photographed image, an extracted region image, a labeled extracted region image, an extracted region model, etc.) required for performing annotation processing, region extraction processing, labeling processing, label change processing, etc.

[0022] The first user terminal 20 according to this embodiment also includes a UI unit 210. The UI unit 210 is realized, for example, by processing executed by a processor such as a CPU by one or more programs (for example, a web browser, etc.) installed in the first user terminal 20.

[0023] The UI unit 210 displays various screens (for example, annotation screens and correction screens, which will be described later) on a display device such as a display etc. The UI unit 210 also accepts various user operations on these various screens.

[0024] The second user terminal 30 according to this embodiment also includes a UI unit 310. The UI unit 310 is realized, for example, by a process executed by a processor such as a CPU by one or more programs (e.g., a web browser, etc.) installed in the second user terminal 30.

[0025] The UI unit 310 displays various screens (for example, a label change screen, which will be described later) on a display device such as a display etc. The UI unit 310 also accepts various user operations on these various screens.

[0026] The overall configuration of the data creation system 1 shown in Fig. 1 is an example and is not limited to this. For example, the data creation system 1 shown in Fig. 1 includes one first user terminal 20 and one second user terminal 30, but there may be multiple of these. Furthermore, for example, the data creation device 10 may be configured with multiple general-purpose servers or the like.

[0027] The annotation process, the region extraction process, the labeling process, and the label change process will be described below.

[0028] [Annotation Processing] The annotation process will be described below.

[0029] <Example of detailed functional configuration of annotation processing unit 110> An example of a detailed functional configuration of the annotation processing unit 110 according to this embodiment is shown in Fig. 2. As shown in Fig. 2, the annotation processing unit 110 according to this embodiment includes a UI providing unit 111, an operation receiving unit 112, an image creating unit 113, and a data saving unit 114.

[0030] The UI providing unit 111 provides an annotation screen (described later) to the first user terminal 20. The operation accepting unit 112 accepts an operation to create a first extracted region image on the annotation screen. The image creating unit 113 creates a first extracted region image from the annotation for the captured image in response to the operation to create the first extracted region image. The data saving unit 114 saves the first extracted region image in the memory unit 150.

[0031] <Annotation process flow> The flow of the annotation process according to this embodiment will be described below with reference to FIG.

[0032] First, the UI providing unit 111 of the annotation processing unit 110 provides an annotation screen to the first user terminal 20 (step S101). As a result, the annotation screen is displayed by the UI unit 210 of the first user terminal 20.

[0033] An example of the annotation screen is shown in Fig. 4. The annotation screen 1000 shown in Fig. 4 includes a photographed image 1100 and a create button 1200. The photographed image 1100 is, for example, a photographed image designated by a first user as an annotation target. The photographed image 1100 shown in Fig. 4 includes seeds 1101 to 1109. The create button 1200 is a button for starting the creation of a first region extraction image.

[0034] The first user can perform annotation operations on the seeds 1101 to 1109 by filling in the areas corresponding to the seeds 1101 to 1109 in the captured image 1100 included in the annotation screen 1000 shown in Fig. 4. Such filling-in operations (annotation operations) can be efficiently performed using, for example, a stylus pen or a touch pen. These annotation operations are accepted by the UI unit 210, and the filled-in areas are reflected on the annotation screen. The annotation operation may be any operation for distinguishing the area to be operated from other areas, and is not necessarily limited to a filling-in operation.

[0035] As an example, Fig. 5 shows the annotation screen 1000 when a fill-in operation has been performed on the seeds 1101 to 1107 in the photographed image 1100 shown in Fig. 4. The photographed image 1100 included in the annotation screen 1000 shown in Fig. 5 includes filled-in areas 1111 to 1117 in which the seeds 1101 to 1107 have been filled in, respectively. Note that in the example shown in Fig. 5, the annotation operation has not yet been performed on the seeds 1108 to 1109, and they have not been filled in.

[0036] The first user can perform the operation of creating a first extracted region image by filling in the areas corresponding to the seeds 1101 to 1109 in the photographed image 1100 and then pressing the create button 1200. The following description will be given assuming that the operation of creating a first extracted region image is performed after filling in the areas corresponding to the seeds 1101 to 1109 in the photographed image 1100. Note that the filling operation does not need to completely fill in the areas corresponding to the seeds; for example, the interior of the areas corresponding to the seeds may not be filled in. The filling operation may also be referred to as, for example, a painting operation or a coloring operation. Furthermore, the color used in the filling operation is not limited, but is preferably a color different from the background color.

[0037] The operation receiving unit 112 of the annotation processing unit 110 receives an operation for creating a first extracted region image (step S102).

[0038] Next, the image creation unit 113 of the annotation processing unit 110 creates a first region extraction image from the captured image on which the annotation operation has been performed (step S103). Here, the image creation unit 113 creates the first region extraction image, for example, by the following steps 11 to 13.

[0039] Step 11: The image creation unit 113 extracts the fill-in area in the captured image by performing a process to extract the difference between the fill-in area and the background of the captured image. This can be achieved using existing image processing technology.

[0040] Step 12: Next, the image creation unit 113 extracts a group of xy coordinates that represent the outline of each fill-in area extracted in step 11.

[0041] Step 13: Then, the image creation unit 113 creates a first extracted region image by associating the captured image with each xy coordinate group extracted in step 12. Note that the image creation unit 113 may assign numbers (identifiers) to the contours represented by the xy coordinate groups in order, and then create data by further associating these numbers. Alternatively, for example, the first extracted region image may be created by further associating each filled-in region.

[0042] An example of the first extracted region image is shown in FIG. 6. The first extracted region image 2000 shown in FIG. 6 includes seeds 1101-1109, and includes contours 1131-1139 of regions corresponding to the seeds 1101-1109, respectively. Each of these contours 1131-1139 is represented by a group of xy coordinates extracted in step 12 above. The first extracted region image 2000 shown in FIG. 6 also includes numbers 1141-1149 assigned to each of the contours 1131-1139 in order from the upper left to the lower right. These numbers 1141-1149 may be assigned in step 13 above, or may be assigned manually by the first user on the annotation screen, for example.

[0043] Then, the data saving unit 114 of the annotation processing unit 110 saves the first region extraction image created in the above step S103 in the storage unit 150 (step S104).

[0044] It should be noted that, when creating and saving a plurality of first extracted region images, the above steps S101 to S104 can be repeated.

[0045] As described above, in the annotation process according to this embodiment, the first user can perform an annotation operation to extract the outline of each seed by simply filling in an area corresponding to each seed in the captured image with a touch pen, stylus pen, or the like. This allows for more efficient annotation than, for example, manually plotting pixels corresponding to the outline of each seed in the captured image one by one. Furthermore, compared to, for example, manually plotting pixels corresponding to the outline of each seed in the captured image one by one, it is possible to eliminate personal dependencies such as dexterity, and it is possible to create uniformly high-quality first extracted region images.

[0046] [Area extraction processing] The region extraction process will be described below.

[0047] <Detailed Functional Configuration Example of Area Extraction Processing Unit 120> An example of a detailed functional configuration of the region extraction processing unit 120 according to this embodiment is shown in Fig. 7. As shown in Fig. 7, the region extraction processing unit 120 according to this embodiment includes a UI providing unit 121, an operation accepting unit 122, a data acquiring unit 123, a relearning unit 124, a region extraction unit 125, a data saving unit 126, and a corrected image creating unit 127.

[0048] The UI providing unit 121 provides a correction screen (described later) to the first user terminal 20. The operation accepting unit 122 accepts an operation to create a third region extraction image (described later) on the correction screen. The data acquiring unit 123 acquires a region extraction image for re-learning a given region extraction model, or acquires a given target image. The re-learning unit 124 re-learns a given region extraction model using the region extraction image for re-learning. The region extraction unit 125 creates a second region extraction image from the target image using the re-learned region extraction model. The data saving unit 126 saves the second region extraction image in the memory unit 150. The corrected image creating unit 127 creates a region extraction image (hereinafter also referred to as a third region extraction image) by correcting the region extraction image to be corrected in accordance with the operation to create the third region extraction image. Here, the correction screen is a screen for correcting the extraction results of the seed region using the re-learned region extraction model (for example, correcting extraction omissions or erroneous extractions).

[0049] <Area extraction process flow (Example 1)> The flow of Example 1 of the region extraction process according to this embodiment will be described with reference to FIG.

[0050] First, the data acquisition unit 123 of the region extraction processing unit 120 acquires a first region extraction image from the storage unit 150 as a region extraction image for re-learning a given region extraction model (step S201).

[0051] Next, the re-learning unit 124 of the region extraction processing unit 120 re-learns the region extraction model by a known supervised learning method using the first region extraction image acquired in step S201 (step S202). That is, the re-learning unit 124 uses the photographed image included in the first region extraction image as input and re-learns the region extraction model so that the contours (x-y coordinate groups representing) of each seed region associated with the photographed image can be accurately extracted. This results in a re-learned region extraction model. Note that re-learning may also be called, for example, model correction (or simply correction), or may be called transfer learning, for example, if the region extraction model before re-learning is a region extraction model for an item other than seeds or a region extraction model for a type or variety of seeds different from the current extraction target.

[0052] The data acquisition unit 123 of the region extraction processing unit 120 acquires the given target image (step S203).

[0053] Next, the region extraction unit 125 of the region extraction processing unit 120 creates a second region extraction image from the target image acquired in step S202 above using the region extraction model after re-learning (step S204). That is, the region extraction unit 125 extracts the contour (a group of xy coordinates representing) of the seed region in the target image using the region extraction model, and then creates the second region extraction image by associating the group of xy coordinates representing the contour with the target image.

[0054] Then, the data saving unit 126 of the region extraction processing unit 120 saves the second region extraction image created in step S204 in the storage unit 150 (step S205).

[0055] The above steps S203 to S205 are repeatedly executed for each given target image.

[0056] <Area extraction process flow (Example 2)> The flow of Example 2 of the region extraction process according to this embodiment will be described with reference to Fig. 9. Note that steps S301 to S305 in Fig. 9 are similar to steps S201 to S205 in Fig. 8, respectively, and therefore description thereof will be omitted.

[0057] Following step S305, the re-learning unit 124 of the region extraction processing unit 120 determines whether or not to re-learn the region extraction model (step S306). Here, the re-learning unit 124 may determine to perform re-learning, for example, when an instruction to perform re-learning is given by the first user. However, this is just one example, and the re-learning unit 124 may also determine to perform re-learning unconditionally, or when some condition is satisfied.

[0058] If it is not determined in step S306 above that re-learning is to be performed, the region extraction processing unit 120 ends the region extraction processing. On the other hand, if it is determined in step S306 above that re-learning is to be performed, the data acquisition unit 123 of the region extraction processing unit 120 acquires second region extraction images from the storage unit 150 as region extraction images for re-learning the region extraction model (step S307). As a result, in step S302, the region extraction model is re-learned using these second region extraction images. This re-learning is expected to result in a region extraction model with higher accuracy than that of Example 1.

[0059] <Area extraction process flow (Example 3)> The flow of Example 3 of the region extraction process according to this embodiment will be described with reference to Fig. 10. Note that steps S401 to S405 in Fig. 10 are similar to steps S201 to S205 in Fig. 8, respectively, and therefore description thereof will be omitted.

[0060] Following step S405, the re-learning unit 124 of the region extraction processing unit 120 determines whether or not to re-learn the region extraction model (step S406). Here, the re-learning unit 124 may determine to re-learn when, for example, the first user issues an instruction to perform re-learning, or may determine to re-learn when the accuracy of the region extraction model is equal to or lower than a predetermined threshold. However, this is just one example, and other than this, for example, the re-learning may be determined to be performed unconditionally, or may be determined to be performed when some condition is satisfied.

[0061] If it is not determined in step S406 above that re-learning should be performed, the region extraction processing unit 120 ends the region extraction processing. On the other hand, if it is determined in step S406 above that re-learning should be performed, the UI providing unit 121 of the region extraction processing unit 120 provides the first user terminal 20 with a correction screen for correcting extraction omissions or erroneous extractions of the second region extraction image (step S407). As a result, the correction screen is displayed by the UI unit 210 of the first user terminal 20.

[0062] An example of the correction screen is shown in FIG. 11. The correction screen 3000 shown in FIG. 11 includes a second extracted region image 3100 and a create button 3200. The second extracted region image 3100 is the extracted region image created and saved in steps S403 and S404. The second extracted region image 3100 shown in FIG. 11 includes seeds 3101 to 3109. It also includes contours 3111 and 3113 to 3119 corresponding to the seeds 3101 and 3103 to 3109, respectively, as well as numbers 3121 and 3123 to 3129 assigned to the contours 3111 and 3113 to 3119, respectively. Meanwhile, the contour of seed 3102 has not been extracted, resulting in an extraction omission. Furthermore, a contour 3511 of an object 3501 other than a seed, such as dirt or dust, has been extracted, and its number 3521 has also been assigned, resulting in an erroneous extraction. The create button 3200 is a button for starting the creation of a third extracted region image.

[0063] The first user can correct an extraction omission by, for example, performing an annotation operation (fill-in operation) on the seed 3102. The first user can also correct an erroneous extraction by performing an operation (deletion) to delete the contour 3511 or the number 3521. After correcting the extraction omission or erroneous extraction, the first user can perform an operation to create a third extracted region image by correcting the second extracted region image (a third extracted region image creation operation) by pressing the create button 3200. The following description will be given assuming that an extraction omission or erroneous extraction has been corrected and then an operation to create a third extracted region image has been performed. The annotation operation and deletion operation are accepted by the UI unit 210, and the filled-in area or deletion is reflected on the correction screen.

[0064] When correcting an extraction omission, the UI unit 210 may, for example, suppress the display of the outlines and the image areas inside them. For example, in the example shown in Fig. 11, the display of outlines 3111, 3113 to 3119, and 3511 and the image areas inside these outlines may be suppressed (i.e., hidden). This makes it possible to display only the seeds that were not extracted (seed 3102 in the example shown in Fig. 11), allowing the first user to easily identify the seeds that were not extracted.

[0065] On the other hand, when correcting an erroneous extraction, the UI unit 210 may, for example, suppress the display of image areas other than the contours and their interiors. For example, in the example shown in FIG. 11, the display of image areas other than contours 3111, 3113 to 3119, and 3511 and the image areas inside these contours may be suppressed (i.e., hidden). This allows the first user to easily grasp the erroneously extracted object (in the example shown in FIG. 11, object 3501, its contour 3511, and number 3521). At this time, the display of the seeds corrected by correcting the omission of extraction may also be suppressed.

[0066] The operation receiving unit 122 of the region extraction processing unit 120 receives an operation to create a third extracted region image (step S408).

[0067] Next, the corrected image creation unit 127 of the region extraction processing unit 120 creates a third region extraction image from the second region extraction image on which the annotation operation or deletion operation (i.e., the operation to correct the extraction omission or incorrect extraction) has been performed (step S409). Here, the corrected image creation unit 127 creates the third region extraction image by performing the following (1) if the operation to correct the extraction omission has been performed, the following (2) if the operation to correct the incorrect extraction has been performed, and both the following (1) and (2) if the operation to correct the extraction omission and the operation to correct the incorrect extraction have both been performed.

[0068] (1) The corrected image creating unit 127 extracts a group of xy coordinates representing the outline of the seed on which the correcting operation has been performed, in the same manner as in steps 11 and 12 above, and newly associates this group of xy coordinates with the second extracted region image.

[0069] (2) The corrected image creating unit 127 deletes the xy coordinate group and its number representing the contour of the object for which the correcting operation has been performed from the second extracted region image.

[0070] After performing (1) or (2) or both of the above, the corrected image creation unit 127 may re-assign numbers (identifiers) to each contour (a group of xy coordinates representing each contour) included in the third region extraction image.

[0071] Next, the data acquisition unit 123 of the region extraction processing unit 120 acquires third region extraction images as region extraction images for re-learning the region extraction model (step S410). As a result, in step S402, the region extraction model is re-learned using these third region extraction images. This re-learning is expected to result in the acquisition of a region extraction model with higher accuracy than that of the second embodiment.

[0072] [Labeling Process] The labeling process will now be described.

[0073] <Example of detailed functional configuration of labeling processing unit 130> An example of a detailed functional configuration of the labeling processing unit 130 according to this embodiment is shown in Fig. 12. As shown in Fig. 12, the labeling processing unit 130 according to this embodiment includes a UI providing unit 131, an operation receiving unit 132, a data acquiring unit 133, a labeling unit 134, and a data saving unit 135.

[0074] The UI providing unit 131 provides the first user terminal 20 with a labeling screen for assigning labels indicating the germination state to seed regions in the region extraction image. The operation accepting unit 132 accepts an operation to create a labeled region extraction image on the labeling screen. The data acquiring unit 133 acquires the region extraction image to be labeled from the storage unit 150. The labeling unit 134 creates a labeled region extraction image from the region extraction image to be labeled. The data saving unit 135 saves the labeled region extraction image in the storage unit 150.

[0075] <Labeling process flow> The flow of the labeling process according to this embodiment will be described with reference to FIG.

[0076] First, the data acquisition unit 133 of the labeling processing unit 130 acquires an area extraction image to be labeled from the storage unit 150 (step S501). For example, the labeling processing unit 130 may acquire a second area extraction image as the area extraction image to be labeled. However, this is not limitative, and the labeling processing unit 130 may acquire any area extraction image as the area extraction image to be labeled.

[0077] Next, the labeling unit 134 of the labeling processing unit 130 assigns a label representing the germination state to each seed region in the region extraction image acquired in step S501 (step S502). Here, the labeling processing unit 130 may assign labels, for example, according to the following (a) or (b). The germination state may be, for example, a three-value state of "normal / abnormal / other," or a two-value state such as "normal / abnormal," "germinated / other," or "germinated / not germinated," or may represent any other classification.

[0078] (a) Human labeling The UI providing unit 131 of the labeling processing unit 130 provides a labeling screen to the first user terminal 20. As a result, the labeling screen is displayed by the UI unit 210 of the first user terminal 20. Thereafter, the first user performs a designation operation to designate a seed region and its label on the labeling screen, and then performs an operation to create a labeled region extraction image. Then, when the operation accepting unit 132 accepts the operation to create a labeled region extraction image, the labeling unit 134 associates a group of xy coordinates representing the contour of the seed region with the label, using the seed region and label specified by the designation operation for the seed region and its label. As a result, a labeled region extraction image in which a label is associated with each contour included in the region extraction image is obtained.

[0079] On the labeling screen, the value of the label may be changed for a seed region that has already been labeled.

[0080] (b) Automatic labeling If a classification model exists that takes an image representing a seed region as input and outputs a label representing the germination state, the labeling unit 134 uses this classification model to label each seed region in the region extraction image to be labeled (for example, labeling the seed regions in the order of the numbers assigned to the contours of the seed regions). At this time, the labeling unit 134 may perform image processing, such as extracting an image region including each seed region from the region extraction image, and then input this image region into the classification model as a single image. This results in a labeled region extraction image in which a label is associated with each contour included in the region extraction image. Note that the above classification model does not necessarily have to be a highly accurate classification model; for example, a classification model for classifying the germination state of seeds of other varieties may be used.

[0081] Note that (a) may be performed after (b), which allows manual correction of label values, especially when the accuracy of the classification model used in (b) is low.

[0082] Then, the data storage unit 135 of the labeling processing unit 130 stores the labeled region extraction image created in step S502 in the storage unit 150 (step S503).

[0083] [Label change process] The label change process will be described below.

[0084] <Detailed Functional Configuration Example of Label Change Processing Unit 140> An example of a detailed functional configuration of the label change processing unit 140 according to this embodiment is shown in Fig. 14. As shown in Fig. 14, the label change processing unit 140 according to this embodiment includes a UI providing unit 141, an operation receiving unit 142, a label changing unit 143, and a data saving unit 144.

[0085] The UI providing unit 141 provides a label change screen (described later) to the second user terminal 30. The operation accepting unit 142 accepts an operation to create a labeled region extraction image after label change on the label change screen. The label changing unit 143 creates a labeled region extraction image by changing the label of the labeled region extraction image to be changed. The data saving unit 144 saves the labeled region extraction image after label change in the memory unit 150.

[0086] <Label change process flow> The flow of the label change process according to this embodiment will be described with reference to FIG.

[0087] First, the UI providing unit 141 of the label change processing unit 140 provides a label change screen to the second user terminal 30 (step S601). As a result, the label change screen is displayed by the UI unit 310 of the second user terminal 30.

[0088] An example of the label change screen is shown in FIG. 16. The label change screen 4000 shown in FIG. 16 includes a labeled region extraction image 4100, a label information display field 4200, and a create button 4300. The labeled region extraction image 4100 is the labeled region extraction image to be changed. The labeled region extraction image 4100 shown in FIG. 16 includes seeds 4101 to 4109, and labels are assigned to the regions corresponding to these seeds 4101 to 4109. The label information display field 4200 is a display field that displays information about the labels assigned to the regions corresponding to the seeds 4101 to 4109 (the number of each label value and the total number of labels). The label information display field 4200 shown in FIG. 16 displays the number of label values ​​"normal" (9), the number of label values ​​"abnormal" (0), the number of label values ​​"other" (0), and the total number of label values ​​"normal", "abnormal", and "other" (9). The create button 4300 is a button for starting the creation of a labeled extracted region image after the label has been changed.

[0089] The second user can change the label values ​​assigned to the selected regions by selecting the regions corresponding to the seeds 4101-4109 in the labeled region extraction image 4100 included in the label change screen 4000 shown in FIG. 16 . For example, by clicking on the region corresponding to a seed using a pointing device such as a mouse, the value of the label assigned to the region can be changed from "normal" to "abnormal" or vice versa. This allows the second user to easily change the label values. In particular, for example, the orientation of seeds in a captured image may vary from seed to seed, or seeds may overlap, making it difficult to accurately label the seeds in the labeling process. This allows the second user to easily change the incorrect label values ​​assigned by the labeling process to the correct values.

[0090] The above label value change method is an example and is not limited to this. For example, each time a change operation is performed on an area corresponding to a seed, the label value may transition from "normal" to "abnormal" to "other" to "normal," etc.

[0091] Also, for example, for each seed, there may be a button for fixing the value of the label assigned to the region corresponding to that seed, allowing the second user to confirm the value of the label assigned to the region corresponding to the seed.

[0092] The second user can change the label values ​​as needed and then create a labeled region extraction image after the label change by pressing the create button 3200. The following description will be continued assuming that the label values ​​of at least some of the regions corresponding to the seeds in the labeled region extraction image 4100 have been changed and then the labeled region extraction image after the label change has been created.

[0093] The operation receiving unit 142 of the label change processing unit 140 receives an operation to create a labeled region extraction image after the label change (step S602).

[0094] Next, the label change unit 143 of the label change processing unit 140 creates a labeled extracted region image by changing the label of the labeled extracted region image to be changed (step S603).

[0095] Then, the data storage unit 144 of the label change processing unit 140 stores (overwrites) the labeled region extraction image created in step S603 in the storage unit 150 (step S604). This results in a labeled region extraction image in which the values ​​of the labels assigned to the seed regions included in the original labeled region extraction image have been changed.

[0096] [Variations] A modification of this embodiment will now be described.

[0097] Variation 1 In the above annotation process, annotation operations were performed to extract (the outline of) the seed region, but annotation operations may also be performed to extract, for example, the outline of the germination region. In this case, the outline of the germination region may also be extracted in the above region extraction process. Furthermore, when the outline of the germination region is extracted in the above region extraction process, a labeling process may also be performed at the same time, and extraction may be performed by combining the extraction of the seed region and the germination state.

[0098] Variation 2 After the seed region (outline) is extracted by the region extraction process, an undetected object region may be detected, for example, using image processing technology, etc. In this case, for each detected object region, the first user may be queried as to whether or not the object region is a seed region.

[0099] Variation 3 For example, when correcting an extraction omission on the correction screen 3000 shown in Fig. 11, a group of x and y coordinates representing the contour of the seed region to be corrected may be extracted based on input of some contour coordinates for the seed region. That is, based on input of some contour coordinates for the seed region, other x and y coordinates constituting the contour may be complemented. In addition, at this time, a group of x and y coordinates representing the contour of the seed region may be extracted based on some or all of information distinguishing the seed region to be corrected (target region) from the background.

[0100] Variation 4 For example, the region extraction model may be retrained by using only region extraction images in which extraction omissions or erroneous extractions have been corrected on the correction screen 3000 shown in FIG. 11 as region extraction images for retraining.

[0101] Variation 5 The annotation operation is not limited to an operation using a stylus pen or a touch pen, but may be, for example, an operation using a mouse or another input device.

[0102] Variation 6 The target image is not limited to a two-dimensional image, but may be, for example, a three-dimensional image having three-dimensional coordinate information (for example, an image having a z coordinate representing depth, distance, etc. in addition to x and y coordinates). In this case, the seed region may be extracted taking into consideration depth information (i.e., the z coordinate representing the height of the seed). The x and y coordinate group corresponding to the seed region may be the three-dimensional coordinates with depth information as they are, or the two-dimensional coordinates obtained by projecting the three-dimensional coordinates onto a plane may be used as the x and y coordinate group.

[0103] Variation 7 The present invention is not limited to annotation work for model learning, and can be applied to other purposes. For example, a process may be performed in which the part related to re-learning is removed from the region extraction process shown in Fig. 10. This makes it possible to check for missed or incorrect extractions from the extraction results of the seed region (or the germination region by further applying Modification 1).

[0104] [summary] As described above, the data creation system 1 according to this embodiment can efficiently create learning data for training a classification model that accurately classifies the germination states of seeds in a photographed image through annotation processing, region extraction processing, labeling processing, and label modification processing. Therefore, by using the data creation system 1 according to this embodiment, it is possible to build a classification model that accurately classifies the germination states of seeds in a photographed image at low cost.

[0105] The present invention is not limited to the above-described specifically disclosed embodiments, and various modifications, changes, and combinations with known technologies are possible without departing from the scope of the claims. [Explanation of symbols]

[0106] 1. Data Creation System 10 Data creation device 20. First user terminal 30 Second user terminal 40 Communication Network 110 Annotation Processing Unit 111 UI provision department 112 Operation reception section 113 Image Creation Department 114 Data Storage Unit 120 Area extraction processing unit 121 UI provision department 122 Operation reception section 123 Data Acquisition Unit 124 Re-learning section 125 Region extraction part 126 Data Storage Unit 127 Corrected Image Creation Department 130 Labeling processing unit 131 UI provision department 132 Operation reception section 133 Data Acquisition Unit 134 Labeling section 135 Data Storage Unit 140 Label change processing unit 141 UI provision department 142 Operation reception unit 143 Label Change Section 144 Data Storage Unit 150 Storage section 210 UI section 310 UI section

Claims

1. an area extraction processing unit that extracts one or more object areas from a first image based on a model that extracts object areas representing one or more objects using an input image including the object; and a UI unit that, when displaying the second image, suppresses display of one or more of the target regions included in the second image on a screen for creating a third image for relearning the model by correcting an extraction omission in the second image; and A device having:

2. an area extraction processing unit that extracts one or more object areas from a first image based on a model that extracts object areas representing one or more objects using an input image including the object; and a UI unit that, when displaying the second image, suppresses display of an area other than the one or more target areas included in the second image on a screen for creating a third image for relearning the model by correcting erroneous extraction of the second image; A device having:

3. a step of generating a second image by extracting one or more object regions from a first image based on a model that extracts object regions representing the object using an image including the object as an input; a step of suppressing display of one or more of the target regions included in the second image when displaying the second image on a screen for creating a third image for re-learning the model by correcting an extraction omission in the second image; How a computer runs.

4. a step of generating a second image by extracting one or more object regions from a first image based on a model that extracts object regions representing the object using an image including the object as an input; a step of suppressing display of an area other than the one or more target areas included in the second image when displaying the second image on a screen for creating a third image for relearning the model by correcting erroneous extraction of the second image; How a computer runs.

5. A program that causes a computer to function as the device according to claim 1 or 2.

Images