Training data generation device, training data generation method, and training data generation program
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- KYOCERA CORP
- Filing Date
- 2024-04-03
- Publication Date
- 2026-08-05
Smart Images
Figure 0007901107000001 
Figure 0007901107000002 
Figure 0007901107000003
Abstract
Description
Technical Field
[0001] (Cross-reference to related applications) This application claims the priority of Japanese Patent Application No. 2021-87206 (filed on May 24, 2021), and the entire disclosure of the application is incorporated herein by reference.
[0002] This disclosure relates to a teacher data generation device, a teacher data generation method, and an image processing device.
Background Art
[0003] Conventionally, a device for creating teacher data including labels attached to an image based on the result of segmenting the image using a machine learning model is known (see, for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0004]
Patent Document 1
Summary of the Invention
[0005] A teacher data generation device according to an embodiment of the present disclosure includes an input unit, a control unit, and an output unit. The input unit acquires at least one input image including an image of a recognition target. The control unit executes a first process of generating polygon data along the contour of a portion determined to be the image of the recognition target from a first region of the input image. The control unit executes a second process of setting segments obtained by dividing the input image based on a luminance gradient. The control unit generates corrected polygon data obtained by correcting the polygon data based on the segments set in the second process. The control unit assigns label information to the input image to generate teacher data. The output unit outputs the teacher data.
[0006] A training data generation method according to one embodiment of the present disclosure includes acquiring at least one input image including an image to be recognized. The training data generation method includes performing a first process to generate polygon data along the contour of the portion of the input image determined to be the image to be recognized from a first region of the input image. The training data generation method includes performing a second process to set segments by dividing the input image into regions based on the brightness gradient. The training data generation method includes generating modified polygon data by modifying the polygon data based on the segments set in the second process. The training data generation method includes assigning label information to the input image. The training data generation method includes generating and outputting training data.
[0007] An image processing apparatus according to one embodiment of the present disclosure comprises an input unit and a control unit. The input unit acquires at least one input image including an image to be recognized. The control unit performs a first process of generating polygon data along the contour of the portion of the input image determined to be the image to be recognized from a first region of the input image. The control unit performs a second process of setting segments by dividing the input image into regions based on the brightness gradient. The control unit generates modified polygon data by modifying the polygon data based on the segments set in the second process. [Brief explanation of the drawing]
[0008] [Figure 1] This block diagram shows an example configuration of a training data generation device according to one embodiment. [Figure 2] This figure shows an example of an input image that includes the object to be recognized. [Figure 3] This figure shows an example of a preprocessed image obtained by performing preprocessing on an input image. [Figure 4] This figure shows an example of an initial polygon image generated to represent the initial polygon to be recognized. [Figure 5] This figure shows an example of the operation screen for selecting the initial polygon generation mode. [Figure 6]This figure shows an example of a segment image generated by performing superpixel processing. [Figure 7] This figure shows an example of a segment image in which superpixels have been applied to a specified area. [Figure 8] This figure shows an example of a segment image in which the region to be deleted has been identified from the region included in the polygon to be recognized. [Figure 9] This figure shows an example of a modified polygon image, which is generated by modifying the initial polygon. [Figure 10] This diagram compares the initial polygon with the modified polygon. [Figure 11] This flowchart shows an example of the procedure for generating training data. [Figure 12] This flowchart shows an example of the procedure for generating initial polygons using machine learning inference. [Figure 13] This flowchart shows an example of a procedure for generating initial polygons by foreground extraction using hue data. [Figure 14] This flowchart shows an example of the procedure for generating initial polygons using graph cuts. [Figure 15] This flowchart shows an example of the procedure for performing machine learning to generate initial polygons based on modified polygon data. [Figure 16] This flowchart shows the next steps following the flowchart in Figure 15. [Modes for carrying out the invention]
[0009] When assigning label information to objects using a machine learning model, the accuracy of assigning label information to objects different from those that have been trained may decrease. In other words, the robustness of label assignment may decrease. According to a training data generation device and a training data generation device according to one embodiment of this disclosure, the robustness of label assignment can be improved.
[0010] (Example configuration of training data generation device 10) A training data generation device 10 according to one embodiment of this disclosure creates training data for generating a machine learning model that performs segmentation at the pixel level on image data that includes an image of a recognition target 50 (see Figure 2, etc.), which is image data having pixels. The machine learning model that performs segmentation is also referred to as the first machine learning model. The training data generation device 10 generates information as training data in which polygons representing the contour of the recognition target 50 are associated with at least one input image 40 (see Figure 2, etc.) which includes an image of the recognition target 50.
[0011] The training data generation device 10 may generate training data by, for example, performing the following procedure. The training data generation device 10 performs a first process to generate polygon data along the contours of the parts of the input image 40 that are determined to be the image of the recognition target 50. In the first process, the training data generation device 10 generates an initial polygon 51 (see Figure 4) as the initial value of the polygon data. The training data generation device 10 also performs a second process to set segments 52 (see Figure 6, etc.) obtained by dividing the input image 40 into regions based on the brightness gradient. The training data generation device 10 may perform a superpixel as the second process to add segmentation information to the input image 40. In other words, the training data generation device 10 sets segments 52 for the input image 40 by performing a superpixel. The training data generation device 10 modifies the polygons based on the segments 52 set in the image data and generates modified polygons 55 (see Figure 9). The modified polygons 55 are also called modified polygon data. The training data generation device 10 generates training data by adding label information for the input image 40 to data in which corrected polygons 55 have been generated as polygon data in the input image 40.
[0012] As shown in FIG. 1, the teacher data generation device 10 includes an input unit 12, a control unit 14, and an output unit 16. The input unit 12 receives the input of an input image 40. The control unit 14 acquires the input image 40 from the input unit 12 and generates teacher data based on the input image 40. The output unit 16 outputs the teacher data generated by the control unit 14.
[0013] The input unit 12 includes an interface for receiving the input of the input image 40. The output unit 16 includes an interface for outputting teacher data. The interface may be configured to include a communication device that can communicate wired or wirelessly. The communication device may be configured to communicate by a communication method based on various communication standards. The communication device can be configured by known communication technologies.
[0014] The input unit 12 may be configured to include an input device that receives input from a user, such as information or data. The input device may be configured to include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may be configured to include physical keys. The input device may be configured to include a voice input device such as a microphone.
[0015] The control unit 14 may be configured to include at least one processor to provide control and processing capabilities for executing various functions. The processor may execute a program for realizing various functions of the control unit 14. The processor may be realized as a single integrated circuit. The integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be realized as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be realized based on various other known technologies.
[0016] The control unit 14 may include a storage unit. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or a memory such as a semiconductor memory or magnetic memory. The storage unit stores various types of information. The storage unit stores programs executed by the control unit 14, etc. The storage unit may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the control unit 14. At least a part of the storage unit may be configured separately from the control unit 14.
[0017] In this embodiment, the control unit 14 comprises an image processing unit 141, an initial polygon generation unit 142, a superpixel unit 143, a polygon correction unit 144, a labeling unit 145, and a training data generation unit 146. Each component of the control unit 14 is configured to perform the processing necessary to generate training data. The control unit 14 may include multiple processors, each corresponding to one of the multiple components. Each processor is configured to perform the processing of each component. The control unit 14 may also be configured to perform the necessary processing with a single processor.
[0018] (Example of operation of the training data generation device 10) A specific example of the operation of the training data generation device 10 is described below.
[0019] <Input image> The input unit 12 receives the input image 40, as illustrated in Figure 2, and outputs it to the control unit 14. The input image 40 includes an image of the recognition target 50 and is used for generating training data. The input unit 12 may accept one image as the input image 40, or it may accept two or more images as input.
[0020] <Pre-treatment> The image processing unit 141 of the control unit 14 performs image processing aimed at reducing noise in the input image 40 acquired from the input unit 12 and enhancing the contours of the recognition target 50. The image processing unit 141 may perform processing such as contrast correction, gamma correction, bilateral filtering, or Gaussian filtering. Depending on the content of the acquired input image 40 or the purpose of image processing, the image processing unit 141 selects processing or adjusts processing parameters so that the contours of the recognition target 50 are enhanced. The processing performed by the image processing unit 141 is also called preprocessing. The image obtained by performing preprocessing on the input image 40 is also called a preprocessed image 41. A preprocessed image 41 is illustrated in Figure 3.
[0021] <Initial Polygon Generation> The initial polygon generation unit 142 of the control unit 14 generates initial polygons 51 for the preprocessed image 41, as illustrated in Figure 4. The initial polygons 51 are lines representing the contour of the recognition target 50. The image from which the initial polygons 51 have been generated is also called the polygon generation image 42. If no preprocessing is performed, the initial polygon generation unit 142 generates initial polygons 51 for the input image 40. The process of generating initial polygons 51 is included in the first process. The initial polygon generation unit 142 may perform the processes described below in order to generate the initial polygons 51.
[0022] <<Inference using machine learning>> The initial polygon generation unit 142 may use a pre-trained machine learning model to perform machine learning object detection inference on the input image 40 or pre-processed image 41, and use the output contour information as the initial polygon 51. The machine learning model used for inference to generate the initial polygon 51 is also called the second machine learning model. The initial polygon generation unit 142 may, considering the possibility that the true contour may not be output when the contour of the recognition target 50 is complex, further perform graph cut processing on the polygon generation image 42 using the initial polygon 51 obtained by machine learning inference as a cost function. The initial polygon generation unit 142 may use the data obtained by graph cut processing as the initial polygon 51.
[0023] <<Foreground extraction using hue data>> In contour extraction, it is known to be effective to train a model that specializes in foreground extraction by training multiple recognition targets 50 as a single class and using this model as a pre-trained model. Therefore, the initial polygon generation unit 142 may specify a region in the input image 40 or pre-processed image 41, extract background color data for the specified region, and use the foreground contour obtained using the hue value of the background color data as the initial polygon 51. This method of extracting the foreground is a common technique known in chroma key compositing and the like. When the background of the image has a simple structure, contours can be extracted quickly even when there are multiple foreground objects as recognition targets 50.
[0024] <<Graph Cut>> The initial polygon generation unit 142 may apply a cost function created by the user to a graph cut process and use the resulting data as the initial polygon 51.
[0025] <<Selection of Processing>> The training data generation device 10 may accept input from the user via the input unit 12 to specify which process the initial polygon generation unit 142 should execute. The input unit 12 may include a user interface. For example, the input unit 12 may present the user with a selection screen as shown in Figure 5 and accept input from the user to select a process. Each mode illustrated in Figure 5 may be associated with information specifying which of the above processes to execute, or it may be associated with parameters specified when executing each of the above processes.
[0026] The initial polygon generation unit 142 may generate the initial polygon 51 by any of the following methods: foreground extraction using hue data, graph cutting, or machine learning inference, or a combination thereof. Foreground extraction using hue data can also be described as background removal based on hue information. Machine learning inference can also be described as inference for the detection of the recognition target 50 by a second machine learning model. In the first process, the initial polygon generation unit 142 may generate polygon data based on a predetermined algorithm that includes at least one of background removal based on hue information, graph cutting, and inference for the detection of the recognition target 50 by a second machine learning model.
[0027] The initial polygon generation unit 142 may specify at least a portion of the input image 40 and generate initial polygons 51 within that region. The region specified as the target for generating initial polygons 51 is also referred to as the first region.
[0028] <Superpixels> Superpixel is a known image processing technique that extracts areas with high brightness gradients in an input image 40 and divides the image into multiple regions along its contour lines. The superpixel unit 143 of the control unit 14 performs superpixel on a designated region 53 that includes at least a portion of the input image 40, as illustrated in Figures 6 and 7, and divides it into segments 52. The superpixel unit 143 associates segmentation information, which identifies the boundaries of the generated segments 52, with the image. The image to which the segmentation information is associated is also called a segment image 43. The execution of superpixel is included in the second process.
[0029] The superpixel unit 143 may appropriately set a designated area 53 (see Figure 6) to be targeted for superpixels. The designated area 53 is also called the second area. The superpixel unit 143 may set the designated area 53 to include all of the initial polygons 51 based on the data of the initial polygons 51. For example, in Figure 6, the superpixel unit 143 generates a segment 52 with the range including the four recognition targets 50 set as the designated area 53. If multiple initial polygons 51 are generated, the superpixel unit 143 may set the designated area 53 to include each initial polygon 51 individually. For example, in Figure 7, the superpixel unit 143 generates a segment 52 with the range including each recognition target 50 individually set as the designated area 53. The superpixel unit 143 may accept input from the user to specify a range and set the designated area 53 based on the user's specification. When the superpixel unit 143 automatically sets the designated area 53, it may be possible to set how much larger the designated area 53 should be relative to the area of the initial polygons 51. The superpixel unit 143 can speed up superpixel processing or reduce the processing load by limiting the processing range rather than the entire image.
[0030] <Polygon Correction> The polygon correction unit 144 of the control unit 14 performs actions such as adding segments 52 to the initial polygon 51 or deleting some segments 52 from the initial polygon 51, based on the initial polygon 51. The polygon correction unit 144 corrects the initial polygon 51 based on user operations for areas where the initial polygon 51 does not accurately capture the contour of the recognition target 50, or deletes data of the initial polygon 51 that has not been labeled. For example, as shown in Figure 8, if the initial polygon 51 includes the shadow of the recognition target 50 as the contour of the recognition target 50, the polygon correction unit 144 designates the segment 52 marked with a star as the deletion target area 54 and deletes it from the initial polygon 51. By deleting the deletion target area 54 from the initial polygon 51, the polygon correction unit 144 can generate a corrected polygon 55 that accurately captures the contour of the recognition target 50, as illustrated in Figure 9. The image associated with the information of the corrected polygon 55 is also called the polygon correction image 44. Furthermore, if the initial polygon 51 does not completely include the contour of the object to be recognized 50, that is, if the segment 52 marked with a star forms the contour of the object to be recognized 50, the polygon modification unit 144 may add the segment 52 marked with a star as the initial polygon 51 to generate the modified polygon 55.
[0031] Referring to Figure 10, the initial polygon 51 and the modified polygon 55 are compared. The polygon modification unit 144 can generate a modified polygon 55 that represents a contour closer to the true contour of the recognition target 50 by deleting the area to be deleted 54 within the range enclosed by the initial polygon 51.
[0032] In addition to modifying the initial polygon 51 based on arbitrary pixel or region specifications by the user, the polygon modification unit 144 may also modify the initial polygon 51 based on segmentation information generated by the superpixel unit 143. For example, when an arbitrary pixel value is specified by the user, the polygon modification unit 144 can generate a modified polygon 55 by modifying the segment 52 containing the specified pixel value as the foreground or background. When modifying the initial polygon 51 based on segmentation information, the user operations required to modify the initial polygon 51 can be reduced compared to when the user modifies it by inputting a trace of the true contour line of the recognition target 50. Furthermore, the modification of the initial polygon 51 can be sped up. For example, in Figure 10, by specifying the area to be deleted 54 within the range enclosed by the initial polygon 51 as the background, the modification of the part corresponding to the shadow of the recognition target 50 can be achieved with fewer operations.
[0033] The polygon correction unit 144 may automatically correct the initial polygon 51.
[0034] The labeling unit 145 of the control unit 14 adds label information to the input image 40 or preprocessed image 41 that describes the recognition target 50 whose contour is represented by the initial polygon 51 or modified polygon 55. If the initial polygon 51 or modified polygon 55 has been generated in the input image 40 or preprocessed image 41, the labeling unit 145 adds the label information to the initial polygon 51 or modified polygon 55. The labeling unit 145 may accept label information input from the user and add the label information specified by the user. The labeling unit 145 may also add label information determined by machine learning inference. The labeling unit 145 may add the label information at any timing during the period from when the input image 40 is acquired from the input unit 12 until the modified polygon 55 is generated by the polygon modification unit 144.
[0035] The training data generation unit 146 of the control unit 14 generates training data by associating the input image 40, the data of the modified polygon 55, and the label information, and outputs it to the output unit 16.
[0036] When the control unit 14 obtains multiple input images 40 from the input unit 12, it performs the processes described above for each input image 40 to generate training data.
[0037] The output unit 16 outputs the training data acquired from the control unit 14 to an external device.
[0038] As described above, the training data generation device 10 according to this embodiment can generate training data by generating and modifying the initial polygon 51.
[0039] (Example of a procedure for generating training data) The control unit 14 of the training data generation device 10 may execute a training data generation method that includes the steps of the flowchart illustrated in Figure 11. The training data generation method may be implemented as a training data generation program to be executed by the processor constituting the control unit 14 of the training data generation device 10. The training data generation program may be stored on a non-temporary computer-readable medium.
[0040] The control unit 14 of the training data generation device 10 acquires the input image 40 via the input unit 12 (step S1). The control unit 14 performs preprocessing on the input image 40 (step S2). The control unit 14 selects the generation mode for the initial polygon 51 (step S3). The control unit 14 selects one of the following modes: machine learning inference, foreground extraction using hue data, and graph cut.
[0041] The control unit 14 generates the initial polygon 51 (step S4). The control unit 14 generates the initial polygon 51 in the mode selected in step S3.
[0042] <Inference using machine learning> The control unit 14 executes the steps in the flowchart shown in Figure 12 to generate the initial polygon 51 in machine learning inference mode.
[0043] The control unit 14 acquires a machine learning model (step S11). The control unit 14 uses the machine learning model to perform inference to detect the contours of the recognition target 50 from the input image 40 (step S12).
[0044] The control unit 14 determines whether to perform a graph cut (step S13). If the control unit 14 does not determine whether to perform a graph cut (step S13: NO), it proceeds to step S15. If the control unit 14 determines whether to perform a graph cut (step S13: YES), it performs a graph cut on the input image 40 using the contours detected by the inference as the cost function (step S14).
[0045] The control unit 14 generates an initial polygon 51 based on the contour of the recognition target 50 detected by the inference process (step S15). After executing the procedure in step S15, the control unit 14 terminates the execution of the flowchart in Figure 12 and proceeds to the procedure in step S5 in Figure 11.
[0046] <Foreground extraction using hue data> The control unit 14 executes the steps in the flowchart shown in Figure 13 to generate the initial polygon 51 in foreground extraction mode using hue data.
[0047] The control unit 14 specifies the range to be extracted as the foreground (step S21). The control unit 14 obtains the background color in the specified range as the surrounding hue (step S22). The control unit 14 removes the background (step S23). The control unit 14 generates an initial polygon 51 based on the contour of the foreground extracted by removing the background (step S24). After executing the procedure in step S24, the control unit 14 finishes executing the flowchart in Figure 13 and proceeds to the procedure in step S5 in Figure 11.
[0048] <Graph cut> The control unit 14 executes the steps in the flowchart shown in Figure 14 to generate the initial polygon 51 in graph cut mode.
[0049] The control unit 14 generates a mask (step S31). The control unit 14 performs a graph cut based on the mask (step S32). The control unit 14 determines whether the graph cut is complete (step S33). If the graph cut is not complete (step S33: NO), the control unit 14 returns to the procedure in step S31. If the graph cut is complete (step S33: YES), the control unit 14 generates an initial polygon 51 based on the extraction result of the recognition target 50 by the graph cut (step S34). After executing the procedure in step S34, the control unit 14 finishes executing the flowchart in Figure 14 and proceeds to the procedure in step S5 in Figure 11.
[0050] The control unit 14 performs superpixel processing (step S5). The control unit 14 modifies the polygon based on segmentation information that identifies the segment 52 generated by the superpixel processing (step S6). The control unit 14 assigns label information (step S7). The control unit 14 determines whether there is another input image 40 for generating training data, that is, whether there is the next image data (step S8). If the next input image 40 exists (step S8: YES), the control unit 14 returns to the procedure in step S2 to process the next input image 40. If the next input image 40 does not exist (step S8: NO), the control unit 14 generates training data by associating the polygon data generated in the input image 40 with the label information assigned to the polygon (step S9). After executing the procedure in step S9, the control unit 14 finishes executing the procedure in the flowchart of Figure 11.
[0051] The control unit 14 may perform the label information assignment procedure in step S7 at any time between step S1 and step S6. If the control unit 14 assigns label information before generating the initial polygon 51, it assigns the label information to the input image 40. If the control unit 14 generates the initial polygon 51 using the input image 40 to which the label information has been assigned, it assigns the label information assigned to the input image 40 to the generated initial polygon 51. The control unit 14 may also extract recognition targets 50 that match the label information assigned to the input image 40 and generate initial polygons 51 for the extracted recognition targets 50.
[0052] The control unit 14 may perform the procedure for generating the initial polygon 51 in step S4 of Figure 11 after the procedure for executing the superpixel in step S5. In this case, the control unit 14 can generate the initial polygon 51 based on the segmentation information.
[0053] The control unit 14 may perform superpixel re-processing on the entire image when correcting polygons. For example, if the control unit 14 performs superpixel re-processing on only a portion of the input image 40 in step S5, segmentation information will not be associated with the area outside the superpixel range. Therefore, it is conceivable in various embodiments that the control unit 14 may perform superpixel re-processing on the entire image when correcting polygons.
[0054] (summary) As described above, the training data generation device 10 and training data generation method according to this embodiment reduce the time required to draw polygons representing the contour of the recognition target 50 by generating and modifying the initial polygon 51. In addition, the user operations required to recognize the recognition target 50 as the foreground and generate the initial polygon 51 are reduced.
[0055] Furthermore, the initial polygon 51 is modified based on segmentation information that identifies segments 52, which are set by performing superpixel operation on the input image 40 or preprocessed image 41. In this way, the initial polygon 51 is modified with high accuracy so that the contour represented by the modified polygon 55 approaches the true contour of the recognition target 50. In addition, the time required to modify the initial polygon 51 is reduced.
[0056] Furthermore, by limiting the superpixel processing to the area surrounding the initial polygon 51 or the target of recognition 50, the processing load of the superpixels can be reduced. As a result, computational costs may be reduced.
[0057] As a comparative example, a configuration in which polygon data representing the contour of an object to be recognized (50) is generated manually by the user is conceivable. In this case, the more complex the contour of the object, the greater the amount of work and time required for the user to specify the contour. According to the training data generation device 10 and training data generation method of this embodiment, polygons can be generated with high accuracy without any user intervention by generating and modifying the initial polygon 51. As a result, the amount of work and time required for the user can be reduced.
[0058] As a comparative example, a configuration can be considered in which training data is generated by performing annotation on a series of images using a combination of background subtraction and deep learning. However, while this configuration is effective for a series of images in which the object is moving, it is not useful when the image contains multiple images of different objects, or when there are no background images that do not show the object. According to the training data generation device 10 and training data generation method of this embodiment, polygons can be generated with high accuracy. As a result, robustness to non-identical objects can be improved.
[0059] As a comparative example, a configuration can be considered in which training data is generated by detecting the contour of the foreground from an arbitrary background with high accuracy. However, in this configuration, if an image contains images of multiple objects, the amount of work required to input the shape of the foreground region increases. According to the training data generation device 10 and training data generation method of this embodiment, the input of the shape of the foreground region can be omitted by generating the initial polygon 51. As a result, the amount of work and working time for the user can be reduced.
[0060] As a comparative example, a configuration in which a machine learning model is used to label the segmentation of each pixel in an image can be considered. However, in this configuration, in order to improve the accuracy of label assignment, it is necessary to create highly accurate training data. Creating highly accurate training data incurs work time and costs for preparing the initial training data, as well as computational load and costs for performing training to generate the machine learning model. According to the training data generation device 10 and training data generation method of this embodiment, polygons can be generated with high accuracy without any work by generating and modifying the initial polygon 51. As a result, the amount of work and work time for the user can be reduced.
[0061] (Other embodiments) Other embodiments are described below.
[0062] <Feedback on correction data for initial polygon 51> The training data generation device 10 may sequentially generate training data for each of the multiple input images 40. In this case, the training data generation device 10 may feed back the correction data of the initial polygons 51 in the input images 40 processed in an earlier order to the generation of initial polygons 51 in the input images 40 processed in a later order. By doing so, the accuracy of the initial polygons 51 can be improved. As a result, the workload or computational load required for correcting the initial polygons 51 can be reduced.
[0063] For example, suppose the initial polygon 51 generated from the input image 40 includes the shadow of the object to be recognized 50. In this case, the training data generation device 10 generates a corrected polygon 55 by removing the shadow portion from the initial polygon 51 in the polygon correction unit 144 of the control unit 14. The control unit 14 features the image of the removed shadow portion as corrected data and feeds back the data characterized by the corrected data to the generation of the initial polygon 51 in the input image 40, which is processed in a later order. Based on the data characterized by the image of the shadow portion as corrected data, the control unit 14 can detect the shadow portion from the input image 40 and remove the shadow portion from the beginning when generating the initial polygon 51.
[0064] The data characterized by the modified image, including the image of the modified parts such as shadows, includes, for example, image pixel value information, texture information, or shape information. The data characterized by the modified image of the initial polygon 51 is also called feature data. The feature data can also be used as a condition for selecting the type of image processing or determining the image processing parameters in the image processing unit 141 of the control unit 14. In other words, the image processing unit 141 may modify the parameters to be applied to the preprocessing of the input image 40 to be processed later, based on the modified polygon data. The parameters applied to the preprocessing are also called preprocessing parameter values.
[0065] When creating training data, the control unit 14 may process multiple input images 40 containing images of the same type of object. When the control unit 14 processes multiple input images 40 containing images of the same type of object to generate an initial polygon 51 using machine learning inference, a machine learning model that is overfitted to the input images 40 may be used as the machine learning model for inference. For example, when starting to process the next input image 40, if predetermined conditions are met, the control unit 14 may generate and transfer a machine learning model to be used for generating the initial polygon 51 by learning using training data generated based on already processed input images 40.
[0066] The control unit 14 may correct the input image 40 as a preprocessing step for the next input image 40 to be processed, based on the difference between the initial polygon 51 (polygon data) and the corrected polygon 55 (corrected polygon data) from a previously executed processing of the input image 40. Alternatively, the control unit 14 may correct the input image 40 as a preprocessing step for the next input image 40 to be processed, based on the correction data from the initial polygon 51 (polygon data) to the corrected polygon 55 (corrected polygon data) from a previously executed processing of the input image 40.
[0067] The control unit 14 may also feed back the corrected data of the initial polygon 51 as described above by executing a training data generation method that includes the steps of the flowchart illustrated in Figures 15 and 16.
[0068] The control unit 14 acquires the input image 40 via the input unit 12 (step S51). The control unit 14 performs preprocessing on the input image 40 (step S52). The control unit 14 selects the generation mode for the initial polygon 51 (step S53). The control unit 14 generates the initial polygon 51 (step S54). The control unit 14 generates the initial polygon 51 in the mode selected in step S53. In step S54, the control unit 14 may perform the procedure shown in Figure 12, Figure 13, or Figure 14. Each step from S51 to S54 in Figure 15 corresponds to each step from S1 to S4 in Figure 11.
[0069] The control unit 14 automatically corrects the initial polygon 51 (step S55). Specifically, the control unit 14 may correct the initial polygon 51 based on the correction data of the initial polygon 51 when the input image 40 was processed before the input image 40 currently being processed. The control unit 14 does not have to perform the procedure in step S55.
[0070] The control unit 14 performs superpixel (step S56). The control unit 14 modifies the polygon based on segmentation information that identifies the segment 52 generated by the superpixel (step S57). The control unit 14 assigns label information (step S58). The control unit 14 determines whether there is another input image 40 for generating training data, that is, whether there is the next image data (step S59). If there is no next input image 40 (step S59: NO), the control unit 14 generates training data by associating the polygon data generated in the input image 40 with the label information assigned to the polygon (step S60). After executing the procedure in step S60, the control unit 14 finishes executing the procedure in the flowchart of Figure 11. Each step from S56 to S60 in Figure 15 corresponds to each step from S5 to S9 in Figure 11.
[0071] If the next input image 40 exists (step S59: YES), the control unit 14 determines whether polygon modification was performed during the processing of the previous input image 40 (step S61). If no polygon modification was performed during the processing of the previous input image 40 (step S61: NO), the control unit 14 returns to the procedure in step S52 to process the next input image 40. If polygon modification was performed during the processing of the previous input image 40 (step S61: YES), the control unit 14 features the modified data (step S62). The control unit 14 learns the modified data (step S63). The control unit 14 may generate a machine learning model to generate the initial polygon 51 by learning the modified data. After executing the procedure in step S63, the control unit 14 returns to the procedure in step S52 to process the next input image 40.
[0072] As described above, the training data generation device 10 can improve the accuracy of the initial polygon 51 by feeding back the feature data. Furthermore, the training data generation device 10 can further improve the accuracy of the initial polygon 51 by automatically correcting the initial polygon 51 based on the feature data. In addition, the training data generation device 10 can enhance the contours of the recognition target 50 and make it easier to detect the initial polygon 51 by adjusting the parameters in the preprocessing of the input image 40 based on the feature data. As a result, the accuracy of the initial polygon 51 is further improved.
[0073] While embodiments of the training data generation device 10 have been described above, embodiments of the device may also include methods or programs for implementing the device, as well as embodiments of a storage medium on which a program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).
[0074] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter; it may also take the form of a program module embedded in an operating system. Moreover, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.
[0075] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are within the scope of this disclosure. For example, the functions and other elements included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated.
[0076] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.
[0077] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.
[0078] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, the first process may swap the identifiers "First" and "Second" with the second process. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers.
[0079] The configuration relating to this disclosure may be implemented as an image processing device comprising: an input unit 12 that acquires at least one input image 40 including an image of the recognition target 50; a first process that generates polygon data along the contour of the portion of the input image 40 determined to be the image of the recognition target 50 from a first region of the input image 40; a second process that sets segments 52 obtained by dividing the input image 40 into regions based on the brightness gradient; and a control unit 14 that generates modified polygon data by modifying the polygon data based on the segments 52 set in the second process. [Explanation of Symbols]
[0080] 10. Training data generation device (12: input unit, 14: control unit, 16: output unit, 141: image processing unit, 142: initial polygon generation unit, 143: superpixel unit, 144: polygon correction unit, 145: labeling unit, 146: training data generation unit) 40 Input Images 41 Preprocessed Images 42 Polygon generation images 43 Segment Image 44 Polygon Correction Image 50 Recognition targets 51 Initial Polygons 52 segments 53 Specified area 54 Delete area 55 Modified Polygons
Claims
1. An input unit that acquires at least one input image containing the image to be recognized, A control unit generates contour data, which is data indicating the contour of the portion of the input image determined to be the target of recognition, and which is different from the data representing the image of the target of recognition, by dividing the input image into multiple segments based on the brightness gradient, selecting the segment that constitutes the target of recognition from among the multiple segments, and generating the contour data as the outer edge of the selected segment. An output unit that outputs the contour data and label information describing the recognition target as training data in association with the input image. A training data generation device equipped with the following features.
2. The training data generation apparatus according to claim 1, wherein in the input image, at least one image of the object to be recognized is separated from other images of the object to be recognized.
3. The training data generation apparatus according to claim 1 or 2, wherein the control unit modifies the contour data by deleting at least one segment from the plurality of segments from the contour data.
4. The training data generation apparatus according to claim 1 or 2, wherein the control unit modifies the contour data by adding at least one segment to the contour data.
5. Obtain at least one input image that includes the image to be recognized, Contour data, which is data indicating the contour of the portion of the input image determined to be the target of recognition, and which is different from the data representing the image of the target of recognition, is generated by dividing the input image into multiple segments based on the brightness gradient, selecting the segment that constitutes the target of recognition from among the multiple segments, and generating the contour data as the outer edge of the selected segment. The contour data and label information describing the recognition target are output as training data in association with the input image. A method for generating training data, including [the specified method].
6. Obtain at least one input image that includes the image to be recognized, Contour data, which is data indicating the contour of the portion of the input image determined to be the target of recognition, and which is different from the data representing the image of the target of recognition, is generated by dividing the input image into multiple segments based on the brightness gradient, selecting the segment that constitutes the target of recognition from among the multiple segments, and generating the contour data as the outer edge of the selected segment. The contour data and label information describing the recognition target are output as training data in association with the input image. A training data generation program that causes the processor to execute.