Learning Data Generation System

Abstract

To provide a learning data generation system, learning data generation device and program which can efficiently perform segmentation of image data used in a learning model such as AI in a short time.SOLUTION: A learning data generation device comprises: a color similarity calculation unit 112 which calculates the color similarity between pieces of image data on the basis of the color feature distribution of the image data obtained by imaging a learning subject; a pattern classification unit 113 which classifies the image data into a plurality of patterns by using the color similarity; a color adjustment unit 122 which corrects a color tone of the image data by using a color adjustment parameter set for each pattern classification in consideration of the imaging environment; a gray scale conversion unit 131 which performs gray scale conversion of the image data subjected to color tone correction; and a region designation unit 133 which performs region designation by sectioning a region in the image data subjected to the gray scale conversion on the basis of a luminance threshold set for each pattern classification.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] An embodiment of the present invention relates to a learning data technology used in AI, machine learning, and deep learning. [Background technology]

[0002] AI can perform highly accurate analysis by segmenting large amounts of related image data. Segmentation (Image Segmentation) is the process of identifying regions of an image that are likely to be meaningful. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-101535 A Summary of the Invention [Problem to be solved by the invention]

[0004] Provided is a learning data generation system that can efficiently and quickly segment image data used in learning models such as AI. [Means for solving the problem]

[0005] The learning data generation system of the embodiment calculates color similarity between image data based on a color feature distribution of the image data obtained by photographing a learning object. a color similarity calculation unit for calculating a color similarity between the first and second pixels; A pattern classification unit that classifies the image data into a plurality of patterns using the color similarity, a color adjustment unit that corrects the color tone of the image data using color adjustment parameters that are set for each pattern classification in consideration of the shooting environment, a conversion unit that converts the color-tone corrected image data into grayscale, and a region designation unit that partitions areas in the grayscale converted image data based on a luminance threshold value that is set for each pattern classification. designation The present invention has a part. [Brief description of the drawings]

[0006] [Figure 1] FIG. 2 is an explanatory diagram of a segmentation function by the training data generation device of the first embodiment. [Diagram 2] FIG. 2 is a functional block diagram of the training data generation device according to the first embodiment. [Diagram 3] 4A to 4C are diagrams for explaining a method of calculating a color similarity in the first function of the first embodiment and pattern classification using the color similarity. [Figure 4] 13A to 13C are diagrams for explaining area designation in the second function of the first embodiment. [Diagram 5] FIG. 4 is a diagram showing a processing flow of a learning data generation process in the first embodiment. BEST MODE FOR CARRYING OUT THEINVENTION

[0007] Hereinafter, an embodiment will be described with reference to the drawings.

[0008] (First embodiment) 1 to 5 are diagrams for explaining a training data generation device 100 according to a first embodiment, and FIG. 1 is a diagram for explaining a segmentation function of the present embodiment.

[0009] The training data generation device 100 of this embodiment generates training data used for AI, machine learning, deep learning, and the like. The training data generation device 100 performs a segmentation process to determine where something is in image data, regardless of whether or not there is a teacher label (answer), and to identify areas in the image that are likely to be meaningful. By using the segmented training data as teacher data, the accuracy of AI, machine learning, and deep learning is improved.

[0010] However, segmentation cannot be done automatically, and the range and area must be specified manually for each image data. For example, when creating training data, a person must manually identify the target part in the image and encircle the target area with dots, and detailed borders must be drawn especially for objects whose shapes change, such as clouds and bubbles. For this reason, segmentation processing requires a huge amount of time and human effort.

[0011] As mentioned above, a large amount of training data is required to improve the accuracy of AI, machine learning, and deep learning. However, generating training data (training data) with the segmentation required for highly accurate learning requires a huge amount of time and human effort, and as a result, the supply of training data has not kept up with demand, resulting in a problem of time-consuming "learning" for AI, machine learning, and deep learning.

[0012] As an example, in water treatment work, there is a stage where purification is performed using microorganisms, and it is very important to manage the state of the microorganisms. Chemicals and air are added to activate the microorganisms, but the area, amount, color, and shape of the bubbles on the water surface change depending on the state of the microorganisms, and it is necessary to control the addition of chemicals and air depending on these changes. It is possible to analyze these with AI and control them automatically, but since a lot of time is spent on image segmentation processing, it is not possible to prepare enough training data necessary for learning. As a result, there have been cases where it was necessary to give up on analyzing with AI and controlling automatically because of the time it took to learn.

[0013] In addition, the shooting environment, i.e., the lighting environment including natural light and / or artificial light, is also a major factor in the manual segmentation process. When the lighting environment changes, the appearance (color) of the learning object changes, the contour becomes unclear, and it becomes difficult to distinguish the learning object from other objects. For example, the appearance of the same object differs between an image in which sunlight is strong and the object appears white overall, and an image in which the sunlight is blocked and the object appears dark overall. In addition, when changes in the area, amount, color, shape, etc. of the object are added, it becomes even more difficult to automate the segmentation process. In other words, the segmentation process necessary for improving accuracy tends to increase the dependency on human region identification and region designation.

[0014] Therefore, the training data generation device 100 of this embodiment automates the segmentation process and generates highly accurate training data quickly and in a short period of time.

[0015] As shown in Fig. 1, the learning data generation process of this embodiment is roughly composed of two functions. The first function calculates a color feature distribution by analyzing the color feature amount of each captured image data, and calculates color similarity by comparing the color feature distributions between image data. Then, pattern classification is performed to group images with similar color similarity.

[0016] The second function performs color adjustment processing according to the shooting environment for each group of images classified into patterns. Then, the images that have been subjected to color adjustment processing are converted to grayscale, and areas within the image data are partitioned and designated using a luminance threshold (brightness threshold) that is set for each group of images classified into patterns.

[0017] The first function classifies patterns using similarities based on the color feature distribution of each image, and can group the appearance (color tone) of the learning target into the same similar group, taking into account the influence of the shooting environment (light environment including natural light and / or artificial light). Therefore, processing suitable for each pattern classification can be performed collectively.

[0018] Next, the second function performs color adjustments according to the shooting environment for each pattern classification all at once, while performing grayscale conversion to identify a specific range of colors within the image (a color range that can be used as the area to be learned), and highlights the area to be specified using a brightness threshold (lightness threshold) set for each pattern classification.

[0019] That is, the second function of this embodiment performs grayscale conversion on the image data to narrow down the color range that can be taken as the learning target area based on the difference in brightness between colors, and for the narrowed-down color range, highlights the contour of the area range based on the brightness threshold. In this way, the contour of the learning target area in the image data is formed based on the brightness threshold, and area designation (area extraction) is performed by dividing the area into areas above the brightness threshold and areas below the brightness threshold, or into areas inside the contour and areas outside the contour.

[0020] Since this is performed collectively for each group of pattern-classified images, it is possible to generate a large amount of image data that has been subjected to segmentation processing, i.e., learning data, in a short period of time.

[0021] 2 is a functional block diagram of the training data generation device 100 of this embodiment. The imaging device captures an image that at least partially includes a learning target (monitoring target). The image captured by the imaging device is input to the training data generation device 100. At this time, the imaging device and the training data generation device 100 may be connected via a wireless or wired communication network so that the captured image data can be directly imported from the imaging device, or the captured image data may be stored from the imaging device in an external storage device and the external storage device may be connected to the training data generation device 100 so that the captured image data can be imported.

[0022] The training data generating device 100 includes a first control device 110, a second control device 120, a third control device 130, a segmentation image output unit 140, and a storage device 150.

[0023] The first control device 110 includes an image data receiving unit 111 that receives input of image data obtained by photographing a learning object and stores the data in a storage unit 150, a color similarity calculation unit 112, and a pattern classification unit 113. The first control device 110 corresponds to the first function of the present embodiment described above.

[0024] FIG. 3 is a diagram for explaining a method for calculating color similarity and pattern classification using color similarity. The color similarity calculation unit 112 calculates color similarity between image data based on the color feature distribution of the image data. The color feature distribution is the occurrence frequency distribution of each of hue, saturation (saturation·chroma), and value (value·brightness) in the HSV color space. The image data has different color feature amounts depending on the light environment including natural light and / or artificial light in the shooting environment. For example, the occurrence frequency of each of the hue, saturation, and value of each pixel is calculated to create a histogram, and the histogram (distribution state) is compared to calculate the color similarity between image data.

[0025] The HSV color space is a color space consisting of three components: hue, saturation, and brightness. Hue is the type of color (e.g., red, blue, yellow) and is divided into hues from 0 to 360. Saturation is the vividness of a color, and as the saturation of a color decreases, the grayness becomes more noticeable and the color becomes dull. Brightness is the brightness of a color. Hue in the HSV color space is information with saturation (0 to 100) and brightness (0 to 100) as parameters, and one pixel has color information (pixel value) of, for example, hue "36", saturation "80", and brightness "100".

[0026] By calculating the occurrence frequency of each of the hue, saturation, and brightness and generating a histogram, the color feature distribution can be grasped, and the similarity of the color feature distribution of the image data can be grasped using a known histogram similarity calculation method.

[0027] The pattern classification unit 113 groups (patterns) similar images based on the calculated color similarity, and forms an image data group for each pattern. Image data groups with similar color similarity have a common color feature distribution, for example, due to similar lighting conditions in the shooting environment. The automatic classification by patterning in this embodiment can form a collection of image data with similar color feature distributions.

[0028] Once each pattern classification is formed, it becomes an individual group having a certain degree of dissimilarity to each other, and image data input thereafter is sorted into a group having a certain range of similarity, or if there is no similarity within the certain range to any of the groups, it is sorted into a new pattern.

[0029] Next, the second control device 120 includes a color adjustment parameter setting unit 121 and a color adjustment unit 122. The color adjustment parameter setting unit 121 receives input of color adjustment parameters that are set for each pattern classification in consideration of the shooting environment.

[0030] The color adjustment parameters are parameters that correct the chromaticity strength of the RGB color space based on wavelengths according to the lighting environment, including natural and / or artificial light, in the shooting environment, or the difference in the HSV color space. For example, when comparing a part exposed to sunlight and a part not exposed to sunlight, the RGB elements of G (green) and B (blue) are stronger in the part exposed to sunlight. Therefore, by setting color adjustment parameters that emphasize R (red) and performing color adjustment of the entire image data, the effects of the lighting environment can be reduced.

[0031] The user can set any value for the color adjustment parameters by referring to each image in the image data group for each pattern classification. The color adjustment parameter setting unit 121 can display an image viewing screen, a color adjustment parameter input screen, etc. on a predetermined display and accept color adjustment parameters for each pattern classification through the screen.

[0032] The third control device 130 includes a grayscale conversion unit 131, a brightness threshold setting unit 132, and an area designation unit 133. The grayscale conversion unit 131 converts the color-tone corrected image data into grayscale. The brightness threshold setting unit 132 accepts input of a brightness threshold for each pattern classification. The area designation unit 133 partitions an area in the grayscale converted image data based on the brightness threshold and designates the area.

[0033] FIG. 4 is a diagram for explaining area designation, in which the second control device 120 and the third control device 130 are responsible for the above-mentioned second function.

[0034] As shown in Fig. 4, image data can be converted to grayscale and an outline of a region range can be formed based on a brightness threshold. Then, for example, the region surrounded by the outline created based on the brightness threshold and the other region are expressed in different colors to generate learning data with a specified region (subjected to segmentation processing). In the example of Fig. 4, the dark parts are the extracted regions whose outlines are formed based on the brightness threshold, and the light parts are the other regions.

[0035] For example, if the extracted region has a strong tendency toward white, a contour made up of pixels equal to or greater than the brightness threshold can be formed, whereas if the extracted region has a strong tendency toward black, a contour made up of pixels equal to or less than the brightness threshold can be formed. In other words, the brightness threshold is used differently depending on the relationship between the color of the learning target (monitoring target) and its surroundings, and a contour equal to or greater than the threshold or a contour less than the threshold is formed to specify the region.

[0036] 5 is a diagram showing a process flow of the learning data generation process. When image data captured by a photographing device is input (S101), color similarity is calculated (S102), and the data is sorted into pattern groups having a certain range of similarity based on the color feature distribution (S103).

[0037] Color adjustment parameters can be set for each of the classified pattern classifications (S104), and color adjustment processing is performed collectively for each pattern classification using the set color adjustment parameters (S105). The color-adjusted image data is converted to grayscale (S106), and area designation processing is performed using the grayscale image. Specifically, a luminance threshold value used in the area designation processing can be set collectively for each pattern classification (S107), and the same luminance threshold value is applied to each image of the same pattern classification.

[0038] Based on the brightness threshold, the contour of the learning range in the image data is formed, and the region is designated (region extracted) by dividing the region into a region above the brightness threshold and a region below the brightness threshold, or into a region inside the contour and a region outside the contour (S108). Since a different brightness threshold is set for each pattern classification, region designation can be performed collectively for each pattern classification.

[0039] The learning data that has been subjected to the segmentation process in this manner is accumulated in the storage unit 150. The accumulated learning data is output via the segmentation image output unit 140 to a device for creating teacher data or the like (S109).

[0040] According to this embodiment, by classifying the similarity between images based on the color feature distribution of each captured image data, it is possible to automatically perform classification taking into account the shooting environment (light environment). Then, for each classified pattern, color adjustment processing taking into account the shooting environment is performed collectively, and grayscale conversion is performed to narrow down the color range that can be taken as the learning target area based on the difference in brightness depending on the color.

[0041] For the narrowed-down color range, the contours of the area range are highlighted based on a brightness threshold value set for each pattern classification, and area designation is performed, eliminating the need for manual area designation and enabling highly accurate segmentation to be performed automatically.

[0042] In this way, it is possible to generate a large amount of highly accurate learning data that has been subjected to segmentation processing in a short period of time.

[0043] Although the embodiment has been described above, the training data generation device 100 can be configured as a part of a learning system equipped with a learning model such as AI, machine learning, deep learning, etc., or as a part of a teacher data generation system for providing teacher data to a learning system. For example, the training data generation device 100 becomes a teacher data generation system if it is configured to have a function of selecting image data that has been subjected to segmentation processing and assigning teacher labels.

[0044] The training data generation device 100 is a computer device equipped with a calculation function, a storage function, a communication function, etc., of a server device, etc. The hardware configuration may include a memory (main storage device), operation input means such as a mouse, a keyboard, a touch panel, a scanner, etc., output means such as a printer, an auxiliary storage device (hard disk, etc.), etc.

[0045] Furthermore, each function of the present invention can be realized by a program, and a computer program prepared in advance to realize each function is stored in an auxiliary storage device, and a control unit such as a CPU reads the program stored in the auxiliary storage device into a main storage device, and the control unit executes the program read into the main storage device to cause a computer to operate the functions of each unit of the present invention. On the other hand, each function of the training data generation device 100 can be configured by an individual device, or a computer system can be configured by connecting multiple devices directly or via a network.

[0046] The above program can also be provided to a computer in a state in which it is recorded on a computer-readable recording medium. Examples of computer-readable recording media include optical disks such as CD-ROMs, phase-change optical disks such as DVD-ROMs, magneto-optical disks such as MO (Magnet Optical) and MD (Mini Disk), magnetic disks such as floppy (registered trademark) disks and removable hard disks, and memory cards such as Compact Flash (registered trademark), Smart Media, SD memory cards, and memory sticks. Also included as recording media are hardware devices such as integrated circuits (IC chips, etc.) specially designed and configured for the purpose of the present invention.

[0047] Although the embodiment of the present invention has been described, the embodiment is presented as an example and is not intended to limit the scope of the invention. This new embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0048] 100 Learning data generation device 110 First control device 111 Image Data Reception Department 112 Color similarity calculation unit 113 Pattern Classification Unit 120 Second control device 121 Color adjustment parameter setting section 122 Color adjustment section 130 Third control device 131 Grayscale conversion section 132 Brightness threshold setting unit 133 Area specification part 140 Segmentation Image Output Unit 150 Storage section

Claims

1. a color similarity calculation unit that calculates a color similarity between image data obtained by photographing a learning object based on a color feature distribution of the image data; a pattern classification unit that classifies the image data into a plurality of patterns using the color similarity; a color adjustment unit that corrects the color tone of the image data using color adjustment parameters that are set for each pattern classification in consideration of the shooting environment; A conversion unit that converts the color-corrected image data into grayscale data; a region designation unit that partitions regions in the grayscale converted image data based on a luminance threshold value that is set for each pattern classification, and designates the regions; A learning data generation system comprising:

2. The image data of the learning object has different color features depending on a light environment including natural light and / or artificial light in the shooting environment, The color feature distribution is a distribution of occurrence frequencies of hue, saturation, and brightness, The learning data generation system according to claim 1, characterized in that the color similarity calculation unit calculates the occurrence frequency distribution of each of hue, saturation, and lightness, and calculates the color similarity by comparing the occurrence frequency distributions between image data.

3. the color adjustment parameter is a parameter for correcting the intensity of chromaticity in an RGB color space or the difference in an HSV color space based on a wavelength according to a light environment including natural light and / or artificial light in a shooting environment, The training data generation device according to claim 1 or 2, characterized in that the area designation unit forms a contour of a learning target range in the image data based on the brightness threshold, and designates the area by dividing it into an area above the brightness threshold and an area below the brightness threshold, or into an area inside the contour and an area outside the contour.

4. A program executed by a computer, A first function of calculating a color similarity between image data obtained by photographing a learning object based on a color feature distribution of the image data; A second function of classifying the image data into a plurality of patterns using the color similarity; a third function of correcting the color tone of the image data using color adjustment parameters that are set for each pattern classification in consideration of the shooting environment; A fourth function of converting the color-corrected image data into a grayscale image; A fifth function of partitioning an area in the grayscale converted image data based on a luminance threshold value set for each pattern classification and specifying the area; A program for causing the computer to realize the above.

5. an image data receiving unit that receives input of image data obtained by photographing a learning subject; a color similarity calculation unit that calculates a color similarity between image data based on a color feature distribution of the image data; a pattern classification unit that classifies the image data into a plurality of patterns using the color similarity; a color adjustment unit that receives input of color adjustment parameters that are set for each pattern classification in consideration of the shooting environment, and corrects the color tone of the image data using the set color adjustment parameters; A conversion unit that converts the color-corrected image data into grayscale data; an area designation unit that accepts an input of a luminance threshold value set for each pattern classification, and partitions an area in the grayscale converted image data based on the luminance threshold value to designate the area; a learning data output unit that outputs image data with a specified region; A learning data generation system comprising:

Images