Method, system and computer program

Abstract

To provide a method that associates suitable information with image data.SOLUTION: A method includes: acquiring target image data of a target image that is an image of a processing target; detecting a target area indicative of an image of an attention target object from within the target image by analyzing the target image data; determining an extension area including the target area and an outside portion of the target area; and storing annotation data indicative of annotation information including area information indicative of the extension area into a storage in association with the target image data.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] This specification relates to techniques for associating information with image data. [Background technology]

[0002] Conventionally, machine learning models for processing images have been used. In order to train the machine learning model, image data and information related to the image data (also called metadata) are used. The metadata indicates, for example, information related to an object in an image (such as the position of the object, an area indicating the object, and identification information of the object). The information related to the image data is added by an operator. Adding information to image data is also called annotation. Patent Document 1 proposes a technology for evaluating the accuracy of information added by an operator and having the operator process an image according to the evaluation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-24665 A Summary of the Invention [Problem to be solved by the invention]

[0004] Proper training of a machine learning model requires the association of appropriate information. Appropriate information varies depending on the machine learning model to be trained. Under these circumstances, it has not been easy to associate appropriate information with image data.

[0005] This specification discloses techniques for associating appropriate information with image data. [Means for solving the problem]

[0006] The technology disclosed in this specification can be realized in the following application examples.

[0007] [Application Example 1] A method for associating information with image data used in training a machine learning model, the method comprising: an acquisition step for acquiring target image data of a target image, which is an image to be processed; a detection step for detecting an object region indicating an image of an object of interest from within the target image by analyzing the target image data; a determination step for determining an extended region including the object region and a portion outside the object region; and an association step for associating annotation data indicating annotation information including region information indicating the extended region with the target image data and storing the annotation data in a storage device.

[0008] According to this configuration, annotation information including area information indicating an extended area including the object area detected by analyzing the target image data and a portion outside the object area is stored in association with the target image data in a storage device, so that an area including other areas in addition to the area indicating the target object of interest can be associated with the target image data.

[0009] The technology disclosed in this specification can be realized in various forms, such as a method and device for identifying information to be associated with image data, an association method and device for associating information with image data, a method and device for generating learning image data, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-temporary recording medium) on which that computer program is recorded, and the like. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating an information processing device according to an embodiment; [Diagram 2] 13A and 13B are schematic diagrams showing examples of captured images. [Diagram 3]1A is a schematic diagram showing an example of the configuration of logo detection model NN1, and FIG. 1B is a schematic diagram showing an overview of the operation of logo detection model NN1. [Figure 4] 13 is a flowchart showing an example of a generation process of a first type data set DS1. [Diagram 5] (A) is an explanatory diagram showing an example of a logo image. (B) is a histogram showing an example of the distribution range of color values. (C) is an explanatory diagram showing divided partial regions. (D) is an explanatory diagram showing an example of a color-changed logo image generated by additional adjustment processing. [Figure 6] 13A-13H are schematic diagrams showing examples of candidate images. [Figure 7] 13 is a flowchart showing an example of a training process for the logo detection model NN1. [Figure 8] 1A is a schematic diagram showing an example of the configuration of the sheet detection model NN2, and FIG. 1B is an explanatory diagram showing an overview of the operation of the sheet detection model NN2. [Figure 9] 13 is a flowchart showing an example of a generation process of a second type data set DS2. [Figure 10] (A) is an explanatory diagram showing an example of a target image. (B) is an explanatory diagram showing an example of a logo region. (C) is an explanatory diagram showing an example of multiple blocks. (D) is an explanatory diagram showing an example of a uniform block. (E) is an explanatory diagram showing an example of an extension region candidate. [Figure 11] 13 is a flowchart illustrating an example of a process for determining candidates for an extension region. [Figure 12] 13A is an explanatory diagram showing an example of a UI screen, FIG. 13B is an explanatory diagram showing an example of a changed contour LAeo, and FIG. 13C is an explanatory diagram showing an example of a UI screen. [Figure 13] 13 is a flowchart showing an example of a training process of the sheet detection model NN2. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] A. First Example: A1. Equipment configuration: FIG. 1 is an explanatory diagram showing an information processing device as an embodiment. In this embodiment, the information processing device 200 is, for example, a personal computer. The information processing device 200 executes various processes for training a machine learning model used to inspect an object (for example, a product such as a printer). The information processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0012] The processor 210 is a device configured to perform data processing, and is, for example, a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores programs 231, 232, 233, and 234, a logo detection model NN1, a first type of data set DS1 for training the logo detection model NN1, a sheet detection model NN2, and a second type of data set DS2 for training the sheet detection model NN2. The models NN1 and NN2 are so-called machine learning models, and are program modules in this embodiment. The programs 231-234, the models NN1 and NN2, and the data sets DS1 and DS2 will be described in detail later.

[0013] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive operations by a user, such as a button, a lever, or a touch panel arranged overlaid on the display unit 240. The user can input various requests and instructions to the information processing device 200 by operating the operation unit 250. The communication interface 270 is an interface for communicating with other devices (for example, a USB interface, a wired LAN interface, or an IEEE802.11 wireless interface). The digital camera 100 is connected to the communication interface 270. The digital camera 100 generates image data of the captured image by capturing an object DV of inspection. Hereinafter, the object DV is assumed to be a printer (the object DV is also referred to as a printer DV).

[0014] A2.Photo: 2(A) and 2(B) are schematic diagrams showing examples of captured images. The first captured image 700x in FIG. 2(A) shows a first printer DVx without any defects. A label sheet 910L (also simply called the sheet 910L) is attached to the first printer DVx. The first captured image 700x includes an image of the sheet 910L. The second captured image 700y in FIG. 2(B) shows a second printer DVy with a defect. The sheet 910L is not attached to the second printer DVy, and the second captured image 700y does not include an image of the sheet 910L. Hereinafter, it will be inspected whether or not a proper label sheet is attached to the printer.

[0015] In this embodiment, the sheet 910L includes a logo image 910. The logo image 910 shows the character string "SAMPLE." In addition to the area of ​​the logo image 910, the sheet 910L also includes other areas (for example, areas showing images of other character strings). Note that the logo is not limited to a character string, and may be various images such as figures, marks, and symbols. Also, the sheet 910L is not limited to a character string, and may be configured using various types of elements (for example, figures, patterns, photographs, and the like).

[0016] The sheet detection model NN2 (Figure 1) is a machine learning model that detects an image of a label sheet (e.g., sheet 910L) from a captured image of the printer using captured image data, which is image data that represents the captured image. If the image of the label sheet is detected, the printer inspection result is a pass. If the image of the label sheet is not detected, the printer inspection result is a fail.

[0017] Image data of various images including an image of a sheet is used for training the sheet detection model NN2. Various information is associated with the data (here, image data) used for training. The process of associating information with data is also called annotation or labeling. Hereinafter, information associated by annotation is also called annotation information. In this embodiment, the annotation information includes area information that specifies an area indicating the sheet to be detected. For example, when image data of the first captured image 700x in FIG. 2(A) is used for training, the annotation information includes area information indicating a frame Fx surrounding the sheet 910L.

[0018] The annotation information is usually determined by an operator. For example, the operator determines a frame Fx surrounding the sheet 910L by observing the first captured image 700x. In addition, various image data are used for training. For example, the display mode of the sheet, such as the position, color, and size of the sheet, may differ between the multiple image data. It is not easy for an operator to determine appropriate annotation information for each of various image data. For example, the operator may determine an inappropriate frame that surrounds only a part of the sheet 910L. Therefore, in this embodiment, the information processing device 200 (FIG. 1) detects a logo image (for example, the logo image 910) from an image for training using the logo detection model NN1. Then, the information processing device 200 determines an extension region that includes a logo region that is an area showing the logo image and a part outside the logo region. The extension region may appropriately show the sheet (for example, the sheet 910L). Then, the information processing device 200 associates annotation information including area information showing the extension region with the image data. Below, the logo detection model NN1 and the sheet detection model NN2 will be explained in that order.

[0019] A3. Logo detection model NN1 configuration: FIG. 3A is a schematic diagram showing an example of the configuration of the logo detection model NN1. In this embodiment, the logo detection model NN1 is an object detection model called YOLO (You only look once). YOLO is disclosed in, for example, the paper "Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788". The YOLO model uses a convolutional neural network to predict a frame containing an object called a bounding box, a confidence that the box contains the object, and a probability (also called a class probability) for each type of object when the box contains the object.

[0020] As shown in Fig. 3(A), the logo detection model NN1 has m (m is an integer of 1 or more) convolutional layers CV11-CV1m, and n (n is an integer of 1 or more) fully connected layers CN11-CN1n following the convolutional layers CV11-CV1m (m is, for example, 24, n is, for example, 2). A pooling layer is provided immediately after one or more of the m convolutional layers CV11-CV1m.

[0021] The convolutional layers CV11-CV1m execute processes including convolution processing and bias addition processing on input data. The convolution processing is a process of sequentially applying s filters of (p×q×r) dimensions to the input data and calculating a correlation value indicating the correlation between the input data and the filters (p, q, r, and s are integers equal to or greater than 1). In the process of applying each filter, multiple correlation values ​​are sequentially calculated while sliding the filter. One filter includes (p×q×r) weights. The bias addition processing is a process of adding a bias to the calculated correlation value. One bias is prepared for each filter. The filter dimensions (p×q×r) and the number s of filters are usually different among the m convolutional layers CV11-CV1m. Each of the convolutional layers CV11-CV1m has a parameter set including multiple weights and multiple biases of multiple filters.

[0022] The pooling layer performs a process to reduce the number of dimensions of data input from the immediately preceding convolution layer. As the pooling process, various processes such as average pooling and maximum pooling can be used. In this embodiment, the pooling layer performs maximum pooling. Maximum pooling reduces the number of dimensions by sliding a window of a predetermined size (e.g., 2×2) with a predetermined stride (e.g., 2).

[0023] The fully connected layers CN11-CN1n output g-dimensional data (i.e., g values, g is an integer equal to or greater than 2) using f-dimensional data (i.e., f values, f is an integer equal to or greater than 2) input from the previous layer. Each of the g output values ​​is a value (inner product + bias) obtained by adding a bias to the inner product of a vector consisting of the input f values ​​and a vector consisting of f weights. The convolution layers CV11-CV1m output g-dimensional data using (f×g) weights and g biases. Note that the number of dimensions f of the input data and the number of dimensions g of the output data are usually different between the n fully connected layers CN11-CN1n. Each of the fully connected layers CN11-CN1n has a parameter set including multiple weights and multiple biases.

[0024] The data generated by each of the convolution layers CV11-CV1m and the fully connected layers CN11-CN1n is input to an activation function and transformed. Various functions can be used as the activation function. In this embodiment, a linear activation function is used in the last layer (here, the fully connected layer CN1n), and a leaky rectified linear unit (LReLU) is used in the other layers.

[0025] FIG. 3B is a schematic diagram showing an overview of the operation of the logo detection model NN1. An image 800 is an example of an input image input to the logo detection model NN1. The input image 800 is represented by the color values ​​of a plurality of pixels arranged in a matrix along a first direction Dx and a second direction Dy perpendicular to the first direction Dx. In this embodiment, the color values ​​are represented by three component values ​​of R (red), G (green), and B (blue). In the example of FIG. 3B, the input image 800 shows two types of logo images 910 and 920. In this embodiment, the first logo image 910 is an image of the character string "SAMPLE". The second logo image 920 is another logo image different from the first logo image 910, and is an image of the character string "SAMPLE2".

[0026] The logo detection model NN1 divides the input image 800 into S×S grid cells 801 (also simply called cells 801) (S is an integer equal to or greater than 2; for example, S is 5). The center of each of the logo images 910, 920 is included in one of the cells 801. The detection result of the logo images 910, 920 (or more generally, the object) is indicated by a prediction value associated with the cell 801 that includes the center of the object region (details will be described later).

[0027] Each cell 801 is associated with Bn rectangular bounding boxes (Bn is an integer equal to or greater than 1, for example, 2). The right-hand part of the middle row of FIG. 3B shows, as examples of bounding boxes, a plurality of first-type bounding boxes BB1c associated with the first logo image 910 and a plurality of second-type bounding boxes BB2c associated with the second logo image 920. The following five predicted values ​​are associated with each bounding box: the center position x in the first direction Dx for the cell 801, the center position y in the second direction Dy for the cell 801, the width w in the first direction Dx, the height h in the second direction Dy, and the confidence level. If the center of the object region is not included in the cell 801, the confidence level of the bounding box associated with the cell 801 is expected to be zero. If the center of the object region is included in the cell 801, the confidence level of the bounding box associated with the cell 801 is expected to be high. Specifically, the confidence is expected to be the same as the intersection over union (IOU) between the bounding box region and the object region, where IOU is the ratio of the area of ​​the intersection of two regions divided by the area of ​​the union of the two regions. Such confidence indicates the degree of agreement between the bounding box and the object region. Note that the confidence is calculated independently of the type of object.

[0028] Here, the logo detection model NN1 detects C types of logo images (C is an integer equal to or greater than 1, for example, C is 3). The types of logo images are also called classes or logo classes. Each cell 801 is further associated with C class probabilities. The C class probabilities correspond to C types of objects (here, logo images). The class probabilities are probabilities under the condition that the center of the object area is included in the cell 801, and indicate the probability for each type of object. Regardless of the total number Bn of bounding boxes associated with one cell 801, one cell 801 is associated with C class probabilities. The left part of the middle row of FIG. 3B shows a class probability map. The class probability map shows the class identifier identified for each cell 801 and corresponds to the class identifier corresponding to the highest class probability. As shown in the figure, in cells 801 close to the first logo image 910, the probability of the class identifier "1", which is the type of the first logo image 910, is high. In cells 801 close to the second logo image 920, there is a high probability of a class identifier of "2", which is the type of the second logo image 920. The multiple first type bounding boxes BB1c on the right side of the drawing are bounding boxes associated with the cell 801 indicating the class identifier of "1" on the class probability map. The multiple second type bounding boxes BB2c are bounding boxes associated with the cell 801 indicating the class identifier of "2" on the class probability map.

[0029] The logo detection model NN1 (FIG. 3(A)) outputs output data 830 indicating S×S×(Bn×5+C) predicted values. Among the S×S×Bn bounding boxes, a bounding box having a certainty equal to or greater than a threshold is adopted as a box (called an object box) indicating a detected object (here, a logo image). In addition, a class identifier corresponding to the highest class probability among the C class probabilities corresponding to the object box is adopted as a class identifier associated with the object box. Note that, as shown in the right part of the middle row of FIG. 3(B), multiple bounding boxes overlapping one logo image may be candidates for the object box. In order to select one bounding box from multiple bounding boxes overlapping each other, a process called "non-maximal suppression" may be performed. This process is a process of deleting one box (for example, a box having a lower certainty) when the IOU between two boxes is equal to or greater than a reference. By repeating this process, one object box corresponding to one logo image is detected. For example, as shown in the lower part of Figure 3(B), a first object box BB1 (class identifier CL1 = 1) representing the first logo image 910 and a second object box BB2 (class identifier CL2 = 2) representing the second logo image 920 are detected.

[0030] A4. Generation process of the first type dataset DS1: 4 is a flowchart showing an example of a process for generating a first type data set DS1 for training the logo detection model NN1. The processor 210 (FIG. 1) executes the process of FIG.

[0031] In S110, the processor 210 acquires logo image data, which is image data of a logo image. In this embodiment, the logo image data is RGB bitmap data, and is stored in advance in the non-volatile storage device 230 (not shown). FIG. 5(A) is an explanatory diagram showing an example of a logo image. In the figure, a first logo image 910 is shown. The first logo image 910 includes a character area 911 and a background area 912. A plurality of pixels in the character area 911 have approximately the same color, and a plurality of pixels in the background area 912 have approximately the same color. The logo image data may be data generated using an image editing application program. Alternatively, the logo image data may be data generated by reading a sample of the logo with a scanner (not shown). In this embodiment, the processor 210 acquires data of a plurality of logo images, including data of the first logo image 910, data of the second logo image 920 (FIG. 3(B)), and data of a third logo image (not shown). Although not shown, both the second logo image 920 and the third logo image, like the first logo image 910, include a text area showing multiple characters and a background area.

[0032] In S115 (FIG. 4), processor 210 clusters the color values ​​of multiple pixels of the logo image, thereby dividing the distribution range of the color values ​​of the logo image into T partial color ranges (T is an integer equal to or greater than 2).

[0033] FIG. 5B is a histogram showing an example of the distribution range of color values. The horizontal axis indicates the brightness value Bv. The range of the brightness value Bv is divided into a number of sections. The vertical axis indicates the number of pixels in each section. This histogram shows the distribution of the brightness value Bv of the first logo image 910 (FIG. 5A). The first light partial color range R1 indicates the distribution range of the brightness value Bv in the character region 911, and the second dark partial color range R2 indicates the distribution range of the brightness value Bv in the background region 912. The processor 210 calculates the brightness value Bv from the RGB color values ​​of each of the multiple pixels, and generates a histogram of the brightness value Bv. Then, when multiple sections each having one or more pixels are consecutive, the processor 210 identifies the range indicated by the consecutive multiple sections as one cluster (i.e., a partial color range). In the example of FIG. 5B, two partial color ranges R1 and R2 are identified.

[0034] In S120 (FIG. 4), the processor 210 divides the logo image into T types of partial regions corresponding to the T partial color ranges. FIG. 5(C) is an explanatory diagram showing the divided partial regions. As shown in the left part of FIG. 5(C), the logo image 910 is divided into a first type region A1 and a second type region A2. The first type region A1 corresponds to the first partial color range R1, i.e., the character region 911, and the second type region A2 corresponds to the second partial color range R2, i.e., the background region 912. Note that one type of partial region corresponding to one partial color range may include multiple regions that are separated from each other, like the first type region A1. Although not shown in the figure, the other logo images are also divided into multiple regions in a similar manner by S115 and S120.

[0035] By S115 and S120, the logo image is divided into T types of partial regions having similar colors. The method of dividing the distribution range of color values ​​into T partial color ranges may be any method of associating a plurality of pixels having similar colors with one partial color range. For example, the range of luminance value Bv may be divided by luminance value Bv corresponding to a valley of the histogram. Furthermore, the distribution range of color values ​​may be divided into T partial color ranges using various color components (e.g., hue, saturation, etc.) other than luminance value Bv. Furthermore, various clustering algorithms such as k-means may be used. The total number T of partial color ranges (i.e., the number T of types of partial regions) is determined for each logo image. Alternatively, T may be determined in advance.

[0036] In S125, the processor 210 generates K (K is an integer equal to or greater than 1) color-changed logo image data by performing an adjustment process that randomly changes the color of one or more types of partial regions. On the right side of FIG. 5(C), three color-changed logo images 910a, 910b, and 910c are shown as examples of color-changed logo images generated from the first logo image 910. The color-changed logo images 910a, 910b, and 910c are different from the original logo image 910 in either one or both of the colors of the first type region A1 and the second type region A2. Although not shown, the processor 210 also generates color-changed logo images from other logo images.

[0037] In this embodiment, the processor 210 changes the color of the entire partial region of one type to the same color that is randomly determined. For example, when the color of the first type region A1 is changed, the colors of all of the multiple characters included in the first type region A1 are changed to the same color.

[0038] The changed color may be close to the original color. For example, when each color value of RGB is expressed in a range of 0 to 255, the color change process may be a process of adding a random value in the range of -100 to +100 to the color value of each color component.

[0039] The processor 210 may also randomly determine the total number of sets of color-changed logo image data to be generated for each logo image. Alternatively, the total number of sets of color-changed logo image data to be generated may be predetermined for each logo image.

[0040] In S130, the processor 210 executes additional adjustment processing of the color-changed logo image data. The additional adjustment processing includes either one or both of a size change processing and an aspect ratio change processing. The size change processing may be either an enlargement processing or a reduction processing. FIG. 5(D) is an explanatory diagram showing an example of a color-changed logo image generated by the additional adjustment processing. In the figure, two color-changed logo images 910a1 and 910a2 generated from a color-changed logo image 910a are shown. The first color-changed logo image 910a1 is an image generated by a size change processing (here, a reduction processing). The second color-changed logo image 910a2 is an image generated by an aspect ratio change processing. The processor 210 also executes additional adjustment processing on color-changed logo images generated from other logo images. The processor 210 randomly determines whether to execute additional adjustment processing, the color-changed logo image to be subjected to additional adjustment processing, and the content of the additional adjustment processing.

[0041] In S135, the processor 210 acquires background image data. The background image data is image data showing a background image on which a logo image is to be arranged. In this embodiment, the processor 210 randomly acquires background image data to be processed from a plurality of background image data (not shown) prepared in advance. The plurality of background image data are stored in advance in the storage device 215 (e.g., the non-volatile storage device 230) (not shown). The plurality of background image data includes data of a background image showing a single-color solid image and data of a photographic background image. A single-color solid image is an image composed of a plurality of pixels having the same color. In this embodiment, each background image is a rectangular image surrounded by two sides parallel to the first direction Dx and two sides parallel to the second direction Dy.

[0042] In S140, the processor 210 generates candidate image data by arranging L logo images (L is an integer equal to or greater than 1) on the background image. The processor 210 selects L logo images from a plurality of logo images including the logo image acquired in S110, the color-changed logo image generated in S125, and the color-changed logo image generated in S130. The processor 210 randomly determines the position of each logo image on the background image. Alternatively, the processor 210 may arrange the logo image at a predetermined position on the background image. In either case, the processor 210 determines the position of each logo image so that the multiple logo images do not overlap each other. The total number L of logo images is determined to a value within a range of 1 or more and the maximum number of logo images that can be arranged on the background image. For example, the processor 210 randomly determines L and randomly selects L logo images.

[0043] Figures 6(A)-6(H) are schematic diagrams showing examples of candidate images. Three candidate images 800a-800c in Figures 6(A)-6(C) include background images 800az, 800bz, and 800cz, and four logo images arranged on the background images 800az, 800bz, and 800cz. The main features of these candidate images 800a-800c are as follows: (I1) Candidate image 800a: The background image 800az is a single-color solid image. (I2) Candidate image 800b: The background image 800bz is a photographic image. (I3) Candidate images 800c: Includes logo images 910 and 910c obtained from the first logo image 910, and logo images 920a and 920b obtained from the second logo image 920. As shown by logo images 920a and 920b in Fig. 6(C), the second logo image 920 is divided into a first area A21 and a second area A22. The logo image 920a is an image generated by changing the color of the second logo image 920. The logo image 920b is an image generated by changing the color of the second logo image 920 and reducing it.

[0044] In S145 (FIG. 4), the processor 210 generates new candidate image data by performing image processing on the candidate image data. This image processing includes one or more processes selected from the group consisting of the following seven processes P1-P7. (P1) Vertical inversion process to invert the candidate image (P2) Horizontal inversion process to invert the left and right sides of the candidate image (P3) Rotation process to rotate the candidate images (P4) A shift process for translating a portion of the color-change target image shown in the region without changing the region showing the color-change target image in the candidate image. (P5) Blurring the candidate image (P6) Noise addition processing to add noise to the candidate image (P7) Color adjustment process to adjust the color of the candidate image

[0045] Five candidate images 800d-800h in Fig. 6(D)-Fig. 6(H) are examples of candidate images generated by the image processing of S145. Candidate image 800f in Fig. 6(F) includes background image 800fz and two logo images 910, 910b arranged on the background image 800fz. Other candidate images 800d, 800e, 800g, 800h include background images 800dz, 800ez, 800gz, 800hz and four logo images arranged on the background images. The main features of these candidate images 800d-800h are as follows: (I4) Candidate image 800d: The background image 800dz is a single-color solid image that has been subjected to left-right inversion processing, and the logo image 910s has been generated by shift processing. (I5) Candidate image 800e: The background image 800ez is a photographic image, and has been subjected to upside-down flipping processing. (I6) Candidate image 800f: has been subjected to a rotation process and a noise addition process of adding noise NZ. (I7) Candidate image 800g: Blurred processing is performed. (I8) Candidate image 800h: Color adjustment processing is being performed.

[0046] In this embodiment, the first direction Dx (FIG. 6(D)) indicates the rightward direction. Therefore, the left-right flipping process flips the position of the first direction Dx. Also, the second direction Dy (FIG. 6(E)) indicates the downward direction. Therefore, the up-down flipping process flips the position of the second direction Dy.

[0047] The shift process (FIG. 6(D)) translates the original logo image to the left within the original region of the logo image 910s. Any part of the logo image after the shift that extends outside the original region of the logo image 910s is deleted. For example, a part of the left side of the first type region A1 is deleted. Furthermore, the color of the blank space 910v that is generated by the translation of the original logo image within the original region of the logo image 910s is set to the same color as the color of the second type region A2 that indicates the background region. The processor 210 randomly determines the direction and amount of movement in the shift process.

[0048] The rotation process (FIG. 6(F)) rotates the original candidate image counterclockwise within the original area of ​​the candidate image 800f. Any part of the rotated candidate image that extends outside the original area of ​​the candidate image 800f is deleted. A copy of a part of the background image 800fz is assigned to a blank area 800fv of the original area of ​​the candidate image 800f that is generated by the rotation of the original candidate image. The processor 210 randomly determines the center of rotation, the direction of rotation, and the angle of rotation.

[0049] The noise addition process (FIG. 6(F)) randomly selects multiple target pixels from the multiple pixels of the candidate image 800f, and adds a random value to the color value of each of the multiple target pixels. Note that the noise addition process may be various other processes. For example, a random value may be added to all pixels of the candidate image. Also, a noise image prepared in advance may be overlaid on the candidate image.

[0050] The blurring process (FIG. 6(G)) is also called a smoothing process. In this embodiment, the blurring process is a process using an average filter, and the entire candidate image 800g is processed. Note that the blurring process may be any process that smoothes color values ​​(for example, other smoothing filters such as a median filter or a Gaussian filter may be used).

[0051] The color adjustment process (FIG. 6(H)) is a gamma correction process that reduces the brightness value in this embodiment, and the entire candidate image 800h is processed. Note that the color adjustment process may be any process that adjusts the color of the candidate image (e.g., a gamma correction process that increases the brightness value, a contrast enhancement process, a saturation adjustment process, a white balance adjustment process, etc.).

[0052] Processor 210 randomly determines whether to execute the image processing of S145, the candidate image to be subjected to the image processing, and the content of the image processing. For example, the processing to be executed is randomly selected from seven processings P1-P7.

[0053] In S150 (FIG. 4), the processor 210 randomly selects Z (Z is an integer equal to or greater than 1) first-type learning image data D11 to be included in the first-type data set DS1 (FIG. 1) from a plurality of candidate image data including the candidate image data generated in S140 and the candidate image data generated in S145 (the number Z is also randomly determined). Then, the processor 210 generates Z label data D12 corresponding to the Z first-type learning image data D11. In this embodiment, the label data D12 is data that determines a target value (i.e., a correct answer) of the output data 830 of the logo detection model NN1 (FIG. 3(A)). Such label data D12 is also called teacher data. Specifically, the label data D12 indicates area information D121 indicating the area of ​​the logo image in the candidate image and a logo class D122 indicating the type of the logo image. The region information D121 indicates the center position of the region in the candidate image (specifically, the position in the first direction Dx and the position in the second direction Dy), the width in the first direction Dx, and the height in the second direction Dy. In this embodiment, the logo images are classified into C classes. The logo class D122 indicates one of the C classes.

[0054] Processor 210 identifies combinations of region information D121 and logo class D122 for each of the L logo images in the candidate image based on the content of the processes in S125-S145. Region information D121 is determined so as to indicate the smallest rectangle that includes the entire logo image. Then, when the candidate image includes L logo images, processor 210 generates label data D12 indicating the L combinations of region information D121 and logo class D122.

[0055] In S155, the processor 210 associates the first type of training image data D11 (FIG. 1) with the labeled data D12 and stores them in the storage device 215 (for example, the non-volatile storage device 230). Hereinafter, the first type of training image data D11 and the labeled data D12 associated with each other are collectively referred to as first type labeled data LD1. The first type data set DS1 includes a plurality of first type labeled data LD1. The processor 210 may store the labeled data LD1 in an external storage device (not shown) connected to the information processing device 200.

[0056] In S160, the processor 210 judges whether a predetermined number of first-type learning image data D11 (i.e., first-type labeled data LD1) has been generated. For proper training of the logo detection model NN1, the total number of each of the C labeled images included in the first-type data set DS1 is set to a large reference value (e.g., 1000) or more. If the total number of any of the C labeled images is less than the reference value (S160: No), the processor 210 proceeds to S125 and generates new labeled data LD1. If the total number of each of the C labeled images is equal to or greater than the reference value (S160: Yes), the processor 210 ends the process of FIG. 4. The generated plurality of labeled data LD1 indicate various images as described in FIG. 6(A)-FIG. 6(H). The first-type data set DS1 includes such a plurality of first-type labeled data LD1. The information processing device 200 is an example of a system that generates a plurality of first-type learning image data D11.

[0057] A5. Training process of logo detection model NN1: FIG. 7 is a flowchart showing an example of the training process of logo detection model NN1 (FIG. 3(A)). Logo detection model NN1 is trained so that output data 830 indicates appropriate region information and an appropriate logo class of the logo image in input image 800. By the training, a plurality of calculation parameters (including a plurality of calculation parameters used in the calculation of each of a plurality of layers CV11-CV1m, CN11-CN1n) used in the calculation of logo detection model NN1 are adjusted. Processor 210 performs the process of FIG. 7 in accordance with second program 232.

[0058] In S210, the processor 210 acquires the first-type data set DS1 from the non-volatile storage device 230. In S220, the processor 210 divides a plurality of labeled data LD1 in the first-type data set DS1 into a training data set and a confirmation data set. For example, the processor 210 adopts 70% of the randomly selected labeled data LD1 as the training data set, and adopts the remaining 30% of the labeled data LD1 as the confirmation data set. Hereinafter, it is assumed that the total number of labeled data LD1 in the training data set is Nt, and the total number of labeled data LD1 in the confirmation data set is Nv (Nt and Nv are both integers equal to or greater than 2).

[0059] In S230, the processor 210 initializes a plurality of calculation parameters of the logo detection model NN1. For example, each calculation parameter is set to a random value.

[0060] In S240, the processor 210 calculates the learning loss using the learning dataset. Specifically, the processor 210 inputs Nt pieces of first type training image data D11 to the logo detection model NN1 to generate Nt pieces of output data 830. Then, the processor 210 calculates the learning loss using the Nt pieces of output data 830 and Nt pieces of label data D12 associated with the Nt pieces of first type training image data D11.

[0061] A loss function is used to calculate the learning loss. The loss function may be any of various functions that calculate an evaluation value of the difference between the output data 830 and the label data D12. In this embodiment, the loss function disclosed in the above-mentioned paper by YOLO is used. This loss function includes the following five components. That is, the loss function includes three components corresponding to the difference in center position, the difference in size (i.e., width and height), and the difference in confidence for a bounding box that should indicate the region of the region information D121. The bounding box that should indicate the region of the region information D121 is the bounding box that has the highest IOU between the region of the region information D121 and the region of the bounding box among the Bn bounding boxes associated with the cell 801 (FIG. 3(B)) that includes the center position of the region information D121. In addition, this loss function includes a component corresponding to the difference between the confidence of the bounding box and an ideal confidence (specifically, zero) for a bounding box that should not correspond to the region of the region information D121. Moreover, this loss function includes a component corresponding to the difference between the C class probabilities and the C correct class probabilities for the cell including the center position of the region information D121. The processor 210 calculates the total value of the Nt losses calculated using the loss function as the learning loss. The learning loss may be various values ​​correlated with the Nt losses, such as the average value or median value of the Nt losses.

[0062] In S250, the processor 210 uses the learning loss to update a plurality of calculation parameters of the logo detection model NN1. Specifically, the processor 210 adjusts the calculation parameters according to a predetermined algorithm so as to reduce the learning loss. As the predetermined algorithm, for example, an algorithm using the backpropagation method and the gradient descent method is used.

[0063] In S260, the processor 210 calculates the confirmation loss using the confirmation dataset. The method of calculating the confirmation loss is the same as the method of calculating the training loss described in S240, except that the confirmation dataset is used instead of the training dataset. Specifically, the processor 210 inputs the Nv pieces of first-type training image data D11 of the confirmation dataset to the logo detection model NN1 having the calculation parameters updated in S250, and generates Nv pieces of output data 830. Then, the processor 210 calculates the confirmation loss using the Nv pieces of output data 830 and the Nv pieces of label data D12 associated with the Nv pieces of first-type training image data D11.

[0064] In S270, the processor 210 determines whether the training is completed. The condition for completing the training may be various conditions. In this embodiment, the condition for completing the training is that both the learning loss and the verification loss are equal to or less than a predetermined reference value. Note that the condition for completing the training may be various conditions that indicate that both the learning loss and the verification loss are small. For example, the reference value of the learning loss may be different from the reference value of the verification loss.

[0065] If the training is not complete (S270: No), the processor 210 proceeds to S240 to continue the training. If the training is complete (S270: Yes), in S280, the processor 210 stores the logo detection model NN1 including the adjusted computation parameters as a trained model in the storage device 215 (here, the non-volatile storage device 230). Then, the processor 210 ends the processing in Fig. 7. The processor 210 may store the logo detection model NN1 in an external storage device (not shown) connected to the information processing device 200.

[0066] The output data 830 from the trained logo detection model NN1 has the following characteristics: A cell including the center of the logo image can indicate a bounding box that properly indicates the area of ​​the logo image and has a high confidence and a proper class probability. In addition, the multiple bounding boxes indicated by the output data 830 can include inappropriate bounding boxes that do not indicate the area of ​​the logo image. Inappropriate bounding boxes are associated with a low confidence. Therefore, the logo image can be properly identified by using the bounding boxes with high confidence.

[0067] As described above, in the generation process of FIG. 4, the processor 210 generates a plurality of first-type learning image data D11 used for training the logo detection model NN1 for detecting a logo, which is an example of an object. Specifically, in S110, the processor 210 acquires logo image data of a logo image, which is an image of a logo. In S115 and S120, the processor 210 divides the logo image into T types of partial regions corresponding to T partial color ranges (T is an integer equal to or greater than 2) obtained by dividing the color distribution range of the logo image. The processor 210 executes an adjustment process including a process (S125) of changing the color of each of the one or more types of partial regions to a color different from the original color. In this way, the processor 210 generates a plurality of color-changed logo image data of a plurality of color-changed logo images. Here, each of the plurality of color-changed logo images is an image of a logo. Also, the plurality of color-changed logo images have the same type of partial regions with different colors. For example, the color-changed logo images 910a and 910b in FIG. 5(C) have the same first type region A1 with different colors. Then, the processor 210 generates candidate image data for the candidate image in S135 and S140. Here, the candidate image data corresponds to the learning image data D11, and the candidate image corresponds to the learning image of the learning image data D11. The processor 210 executes the processes of S125-S140 multiple times. Specifically, the processor 210 generates multiple color-changed logo image data. Then, the processor 210 generates multiple candidate image data for the multiple candidate images by using one or more background image data and multiple color-changed logo image data. Here, the candidate image includes a background image indicated by any of one or more background image data, and one or more color-changed logo images arranged on the background image (FIGS. 6(A)-6(H)). The multiple candidate images include different color-changed logo images from the multiple color-changed logo images generated. For example, the candidate image 800c (FIG. 6(C)) includes a color-changed logo image 910c that is not included in the candidate image 800f (FIG. 6(F)). On the other hand, the candidate image 800f includes a color-changed logo image 910b that is not included in the candidate image 800c. In this manner, the processor 210 can generate a plurality of training image data D11 that show images of the logo expressed in various colors.Such a plurality of training image data D11 can appropriately train a machine learning model that processes images of logos (for example, logo detection model NN1).

[0068] As described in S135 (FIG. 4), FIG. 6(B), etc., the one or more background image data includes background image data of the photographic background image 800bz. Thus, the processor 210 can generate a plurality of learning image data D11 showing an image of a logo on the photographic background image. Such a plurality of learning image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process the image of the logo on the photographic background image. The available plurality of background image data may include a plurality of background image data showing different photographs. The plurality of background images may include various photographs such as landscapes, people, furniture, stationery, etc. Such a plurality of learning image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process the image of the logo regardless of the content of the background image.

[0069] As described in S135 (FIG. 4), FIG. 6(A), etc., the one or more background image data includes background image data of a background image 800az showing a monochromatic solid image. Thus, the processor 210 can generate a plurality of training image data showing an image of a logo on a background image showing a monochromatic solid image. Such a plurality of training image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process an image of a logo on a background image showing a monochromatic solid image. The available plurality of background image data may include a plurality of background image data showing solid images of mutually different colors. Such a plurality of training image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process an image of a logo regardless of the color of the background image.

[0070] It is preferable to generate a plurality of types of training image data D11 having a plurality of types of background images showing different contents, such as a photographic background image and a background image showing a single-color solid image. Such a plurality of types of training image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process images of logos on various background images.

[0071] The processor 210 also performs a step S125 to generate a plurality of color-changed logo image data. (FIG. 4). In the embodiment of FIG. 4, the adjustment process further includes S130. S130 includes one or both of a process of changing the size of the color change object image and a process of changing the aspect ratio of the color change object image. Thus, the processor 210 can generate training image data D11 showing an image of a logo in which one or both of the size and the aspect ratio have been changed. Such training image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process an image of a logo in which one or both of the size and the aspect ratio have been changed.

[0072] As described in S140 (FIG. 4) and FIGS. 6(A)-6(H), the process of generating the training image data D11 includes a process of generating training image data D11 of a training image 800a including a background image 800az and a plurality of color-changed logo images 910b, 910a2, and 910c arranged on the background image 800az. When one piece of training image data D11 indicates a plurality of color-changed logo images, a machine learning model for detecting logo images (e.g., logo detection model NN1) can be trained more efficiently than when one piece of training image data D11 indicates a single color-changed logo image.

[0073] As described in S140 (FIG. 4), FIG. 6(C), etc., the process of generating the learning image data D11 includes a process of generating learning image data D11 of an image 800c including a background image 800cz, one or more color-changed logo images 910c arranged on the background image 800cz, and images 920a, 920b of other logos arranged on the background image 800cz. When one piece of learning image data D11 shows an image of a logo and an image of another logo, a machine learning model that detects logo images (e.g., logo detection model NN1) can be trained more efficiently than when one piece of learning image data D11 shows only an image of the same logo.

[0074] As described in S140 (FIG. 4) and FIGS. 6(A)-6(H), the processor 210 arranges multiple logo images on one training image so that they do not overlap each other. Therefore, the training image data D11 can appropriately train a machine learning model that detects logo images (for example, the logo detection model NN1).

[0075] Also, as described in S145 (FIG. 4), the process of generating the training image data D11 includes a process of generating the training image data D11 by performing image processing on the candidate image data of the candidate image including the background image and one or more color-changed logo images arranged on the background image. Here, the image processing includes one or more processes selected from the group consisting of the above seven processes P1-P7. Thus, the processor 210 can generate the training image data D11 showing the logo represented in various formats. Such training image data D11 can train a machine learning model (e.g., logo detection model NN1) to appropriately process the image of the logo represented in various formats.

[0076] A6. Sheet detection model NN2 configuration: FIG. 8(A) is a schematic diagram showing an example of the configuration of the sheet detection model NN2. In this embodiment, the sheet detection model NN2 is a YOLO model, and its configuration is the same as that of the logo detection model NN1 (FIG. 3(A)). The sheet detection model NN2 has p (p is an integer of 1 or more) convolution layers CV21-CV2p and q (q is an integer of 1 or more) fully connected layers CN21-CN2q following the convolution layers CV21-CV2p (p is, for example, 24, and q is, for example, 2). A pooling layer (for example, a layer that performs max pooling) is provided immediately after one or more convolution layers of the p convolution layers CV21-CV2p. Note that p may be different from m in FIG. 3(A). Also, q may be different from n in FIG. 3(A).

[0077] FIG. 8B is an explanatory diagram showing an outline of the operation of the sheet detection model NN2. An image 700 is an example of an input image input to the sheet detection model NN2. The input image 700 is an image of the printer DV captured, similar to the captured images 700x and 700y in FIG. 2A and FIG. 2B. The input image 700 is represented by the color values ​​of a plurality of pixels arranged in a matrix along a first direction Dx and a second direction Dy perpendicular to the first direction Dx. In this embodiment, the color values ​​are represented by three component values ​​of R (red), G (green), and B (blue). In the example of FIG. 8B, the input image 700 includes an image of a sheet 910L including a first logo image 910.

[0078] Like the logo detection model NN1 in Figures 3(A) and 3(B), the sheet detection model NN2 detects the area of ​​an object image. The difference with the logo detection model NN1 is that the sheet detection model NN2 is trained to detect label sheet images instead of logo images. Also, in this embodiment, C types of label sheets corresponding to C types of logo images are available. There is a one-to-one correspondence between the label sheet types and the logo image types. The sheet detection model NN2 then detects C types of label sheets images. Hereinafter, the label sheet types will also be referred to as sheet classes.

[0079] Although not shown, the sheet detection model NN2 detects a bounding box indicating an image of a label sheet according to the same algorithm as the algorithm of the logo detection model NN1 in Fig. 3(B). In the example of Fig. 8(B), a bounding box BBL indicating the sheet 910L is detected. A class identifier CLL is associated with the bounding box BBL. The class identifier CLL is a class identifier corresponding to the highest class probability among the C class probabilities. A class identifier of "1" indicates the first sheet 910L.

[0080] The sheet detection model NN2 outputs output data 730 indicating S×S×(Bn×5+C) predicted values. Similar to the output data 830 in FIG. 3(A), the output data 730 indicates the area of ​​the image of the object (here, the label sheet) by a bounding box having a confidence level equal to or higher than a threshold. In addition, the class identifier corresponding to the highest class probability among the C class probabilities corresponding to the bounding box is adopted as the class identifier associated with the bounding box.

[0081] A7. Annotation process (generation process of the second type dataset DS2): FIG. 9 is a flowchart showing an example of a generation process of the second type data set DS2 (FIG. 1) for training the sheet detection model NN2. The second type data set DS2 includes a plurality of second type labeled data LD2. The second type labeled data LD2 includes second type learning image data D21 including an image of a label sheet, and label data D22 associated with the second type learning image data D21. The label data D22 indicates area information D221 indicating an area of ​​a sheet image in an image, and sheet class information D222 indicating a type of the sheet image, similar to the label data D12 described in S150 and S155 of FIG. 4. As will be described later, in the process of FIG. 9, the processor 210 executes a process of associating the label data D22 indicating the area information D221 and the label data D22 with the second type learning image data D21 (this process is an example of annotation process). The processor 210 executes the process of FIG. 9 according to the third program 233.

[0082] In S310, the processor 210 acquires target image data, which is image data to be processed. In this embodiment, the processor 210 acquires unprocessed sheet image data as target image data from a plurality of sheet image data prepared in advance. The plurality of sheet image data are stored in advance in the storage device 215 (e.g., the non-volatile storage device 230) (not shown). Each of the plurality of sheet image data indicates an image including a label sheet. As described above, in this embodiment, C type of label sheet corresponding to C type of logo image is available. The plurality of sheet image data includes C type of sheet image data indicating C type of label sheet. FIG. 10(A) is an explanatory diagram showing an example of a target image. The target image 700a includes an area of ​​the image of the first sheet 910L and a background area 700az. The image of the first sheet 910L includes the first logo image 910.

[0083] In this embodiment, the sheet image data is generated by placing an image of a sheet on a background image showing a single-color solid image shown by the background image data. In an actual captured image of the printer, the background area shows the outer surface of the printer. In this embodiment, the color of the outer surface of the printer is the same regardless of the position. Therefore, even in an actual captured image, the image of the background area is approximately a single-color solid image. Note that the background image is not limited to a single-color solid image, and may be various images such as a captured image of the outer surface of the printer. Furthermore, the multiple sheet image data may be generated by photographing a printer having a label sheet with a digital camera.

[0084] In S315 (FIG. 9), the processor 210 identifies a logo region by analyzing the target image data using the logo detection model NN1 (FIG. 3(A)). Specifically, the processor 210 generates output data 830 by inputting the target image data to the logo detection model NN1. Then, the processor 210 adopts a rectangular region surrounded by a bounding box indicated by the output data 830 (specifically, a bounding box having a degree of certainty equal to or greater than a predetermined threshold) as the logo region. FIG. 10(B) is an explanatory diagram showing an example of a logo region. The bounding box BBt indicates the first logo image 910 on the target image 700a. The processor 210 identifies the region surrounded by the bounding box BBt as the logo region LA. The processor 210 also identifies the class identifier associated with the highest class probability among the C class probabilities associated with the bounding box BBt as the logo class CLt indicating the type of the logo region LA (CLt=1 in the example of FIG. 10(B)).

[0085] In S320 (FIG. 9), the processor 210 determines whether or not a logo region has been detected. If a logo region has been detected (S320: Yes), in S325, the processor 210 determines an extension region that includes the logo region.

[0086] Fig. 11 is a flowchart showing an example of a process for determining candidates for an extension region. In S410, the processor 210 divides the target image into a plurality of blocks. Fig. 10(C) is an explanatory diagram showing an example of the plurality of blocks. The target image 700a is divided into a plurality of blocks BL of a predetermined shape. The arrangement of the plurality of blocks BL in the target image 700a is predetermined.

[0087] In S420 (FIG. 11), the processor 210 calculates edge intensity values ​​of each of the blocks BL. The edge intensity value is an evaluation value of the ratio of a change in color to a change in position on the target image. In this embodiment, the processor 210 uses a so-called Laplacian filter to calculate the edge amount of each pixel (for example, the absolute value of the calculation result by the filter). A predetermined color component (for example, a luminance value) is used to calculate the edge amount. The processor 210 then calculates the average value of the edge amounts of the multiple pixels in the block BL as the edge intensity value of the block BL. Note that the edge intensity value may be calculated by various other methods. For example, the filter may be any filter for calculating the edge amount (such as a Sobel filter or a Prewitt filter) instead of the Laplacian filter. Also, the edge intensity value of the block BL may be various values ​​having a correlation with the edge amount of the multiple pixels, such as a median value or a mode, instead of the average value of the edge amounts of the multiple pixels.

[0088] In S430, the processor 210 specifies the blocks BL having edge strength values ​​equal to or less than a predetermined standard as uniform blocks. Hereinafter, among the multiple blocks BL, blocks BL that are different from the uniform blocks are also referred to as non-uniform blocks.

[0089] FIG. 10D is an explanatory diagram showing an example of a uniform block. Among the blocks BL in the target image 700a, the blocks BL1 with hatching are uniform blocks BL1, and the blocks BL2 without hatching are non-uniform blocks BL2. As shown in the figure, the blocks BL in the background area 700az outside the first sheet 910L are uniform blocks BL1. Among the blocks BL representing the first sheet 910L, many blocks BL are non-uniform blocks BL2. Note that some of the blocks BL representing the first sheet 910L are uniform blocks BL1. In general, a label sheet may include other elements such as characters, figures, marks, and symbols in addition to a logo image. Therefore, the proportion of uniform blocks BL1 among the blocks BL representing the label sheet 910L is small. Furthermore, the pixels representing the contour 910Lo of the label sheet 910L have a large edge amount. Therefore, the blocks BL representing the contour 910Lo of the label sheet 910L are likely to be non-uniform blocks BL2.

[0090] In S440 (FIG. 11), the processor 210 adopts the same area as the logo area as the initial area of ​​the extension area. The processor 210 then determines a candidate extension area including the logo area by moving the contour of the extension area toward the outside of the logo area. The processor 210 moves the contour so that the entire contour is included in the uniform block BL1. FIG. 10(E) is an explanatory diagram showing an example of a candidate extension area. The candidate extension area LAe on the target image 700a includes the logo area LA and a portion outside the logo area LA. The entire contour LAeo of the candidate extension area LAe is included in the uniform block BL1. As described above, the block BL showing the contour 910Lo of the label sheet 910L is likely to be a non-uniform block BL2. Therefore, the processor 210 can determine a candidate extension area LAe having a contour LAeo that surrounds the contour 910Lo of the label sheet 910L from the outside. Such a candidate extension area LAe includes the entire label sheet 910L.

[0091] The process of moving the contour may be various processes. In this embodiment, the contour LAo of the logo area LA is composed of four sides (i.e., the top side, bottom side, left side, and right side) that form a rectangle. The processor 210 repeats the process of moving the four sides outward in order by a predetermined amount until the entire four sides are included in the uniform block BL1. This allows the processor 210 to determine a candidate extension area LAe that includes the entire label sheet 910L and is smaller than the target image 700a.

[0092] Upon completion of S440, processor 210 ends the process of FIG. 11, that is, S325 of FIG.

[0093] In S330, the processor 210 selects candidate sheet class information from the C pieces of sheet class information based on the logo class specified in S315. In this embodiment, when the logo class specified in S315 corresponds to one of the two logo images 910 and 920, the processor 210 adopts two pieces of sheet class information corresponding to the two logo images 910 and 920 as candidates. In this embodiment, the C types of available logo images include a third logo image (not shown). Then, when the logo class specified in S315 corresponds to the third logo image, the processor 210 adopts one piece of sheet class information corresponding to the third logo image as a candidate. The correspondence between the logo class specified in S315 and the candidate sheet class information is determined in advance. As the candidate sheet class information, sheet class information that may be appropriate in light of the logo class specified in S315 is adopted. The processor 210 selects the candidate associated with the logo class.

[0094] In S335, processor 210 displays a user interface screen (also referred to as a UI screen) on display unit 240 (FIG. 1). FIG. 12(A) is an explanatory diagram showing an example of a UI screen. This UI screen 600 includes a first user interface image 610 and a second user interface image 620. This UI screen 600 shows an example in which the number of candidate sheet class information selected in S330 (FIG. 9) is two or more.

[0095] The first user interface image 610 is a user interface image for allowing the user to change the position of the contour LAeo of the candidate extension area LAe. The first user interface image 610 shows a target image 700a including a first sheet 910L and the contour LAeo of the candidate extension area LAe. The user can move the contour LAeo by operating the operation unit 250 (FIG. 1).

[0096] The second user interface image 620 is a user interface image for allowing the user to specify sheet class information indicating the classification of the candidate extension area LAe (i.e., the classification of the label sheet). The second user interface image 620 shows a candidate area 621 indicating one or more candidates of sheet class information selectable by the user, and a check box 622 indicating one candidate selected from the one or more candidates. The check box 622 with a solid line indicates a selectable candidate, and the check box 622 with a dashed line indicates a non-selectable candidate. The selectable candidate is the candidate selected in S330.

[0097] In the example of FIG. 12(A), the second user interface image 620 shows four pieces of sheet class information CC1, CC2, CC3, and CC4. The first sheet class information CC1 corresponds to the first logo image 910 (FIG. 3(B)), the second sheet class information CC2 corresponds to the second logo image 920, the third sheet class information CC3 corresponds to a third logo image (not shown), and the fourth sheet class information CC4 shows a barcode. The two pieces of sheet class information CC1 and CC2 are selectable, and the other sheet class information CC3 and CC4 are not selectable. The user can check (i.e., select) one of the one or more selectable candidates by operating the operation unit 250 (FIG. 1). In S335 (FIG. 9), the processor 210 adopts the sheet class information corresponding to the logo class identified in S315 as the default sheet class information. The processor 210 then displays the second user interface image 620 in a state in which the default sheet class information is selected.

[0098] Figure 12(C) shows an example of a UI screen when the number of candidate sheet class information selected in S330 (Figure 9) is 1. The first user interface image 610 shows target image 700c, which includes an image of label sheet 930L. Label sheet 930L includes third logo image 930. The second user interface image 620 shows that the third sheet class information CC3 is selectable, and that the other sheet class information CC1, CC2, and CC4 are not selectable.

[0099] It should be noted that in second user interface image 620, the display of unselectable candidates may be omitted.

[0100] In S340 (FIG. 9), processor 210 accepts a change in the position of contour LAeo by the user. FIG. 12(B) is an explanatory diagram showing an example of a changed contour LAeo. In the example of FIG. 12(B), the user moves each of the four sides of contour LAeo closer to contour 910Lo of label sheet 910L. This allows candidate extended area LAe to properly indicate the area of ​​label sheet 910L. If the position of contour LAeo is changed by the user, processor 210 determines the area having the contour in the changed position as the final extended area. Note that the user can input an instruction to accept the position of contour LAeo without changing it by operating operation unit 250. In this case, processor 210 determines the candidate extended area LAe determined in S325 as the final extended area.

[0101] In S345 (FIG. 9), the processor 210 judges whether the total number of selectable candidates of the sheet class information is 1. If the number of selectable candidates is greater than 1 (S345: No), in S355, the processor 210 accepts the designation of the sheet class information by the user. In the example of FIG. 12(A) and FIG. 12(B), the user can select one of two pieces of sheet class information CC1 and CC2 by operating the operation unit 250 (FIG. 1). For example, the logo class specified in S315 (FIG. 9) may be incorrect. That is, the default sheet class information adopted in S335 may be incorrect. The user can confirm the appropriate sheet class information by observing the label sheet displayed in the first user interface image 610. Then, the user can designate the appropriate sheet class information by operating the operation unit 250. Also, if the default sheet class information is correct, the user can input an instruction to accept the default sheet class information by operating the operation unit 250. After S355, the processor 210 proceeds to S360.

[0102] If the total number of selectable candidates for the seat class information is 1 (S345: Yes), in S350, the processor 210 determines the seat class information to be the candidate seat class specified in S330. Then, the processor 210 proceeds to S360.

[0103] In S360, the processor 210 generates annotation data indicating annotation information including region information indicating the candidate extension region LAe and the sheet class information determined in S350 or S355. In S365, the processor 210 associates the target image data with the annotation data and stores them in the storage device 215 (e.g., the non-volatile storage device 230). The entire set of the target image data and annotation data associated with each other forms second type labeled data LD2 (FIG. 1). The target image data corresponds to the second type learning image data D21, and the annotation data corresponds to the label data D22. The processor 210 may store the labeled data LD2 in an external storage device (not shown) connected to the information processing device 200.

[0104] After S365, the processor 210 proceeds to S370. If no logo area is detected in S315 (S320: No), the processor 210 skips S325-S365 and proceeds to S370. In S370, the processor 210 judges whether or not the processing of all sheet image data is completed. If unprocessed sheet image data remains (S370: No), the processor 210 proceeds to S310 and processes new target image data. If the processing of all sheet image data is completed (S370: Yes), the processor 210 ends the processing of FIG. 9. As a result, the second type data set DS2 is generated. Note that the information processing device 200 is an example of a system that associates the label data D22 with the second type learning image data D21.

[0105] A8. Training process of sheet detection model NN2: FIG. 13 is a flowchart showing an example of a training process of the sheet detection model NN2 (FIG. 8(A)). The sheet detection model NN2 is trained so that the output data 730 indicates appropriate area information and appropriate sheet class information of the image of the label sheet in the input image 700. By the training, a plurality of calculation parameters (including a plurality of calculation parameters used in the calculation of each of the layers CV21-CV2p, CN21-CN2q) used in the calculation of the sheet detection model NN2 are adjusted. The processor 210 performs the process of FIG. 13 according to the fourth program 234.

[0106] The training process of FIG. 13 is the same as the training process of FIG. 7, except that the model to be trained is the sheet detection model NN2, and the data set used for training is the second type data set DS2. S510-S580 of FIG. 13 are the same as S210-S280 of FIG. 7, respectively (detailed description is omitted). The output data 730 from the trained sheet detection model NN2 can indicate a bounding box that appropriately indicates the area of ​​the image of the label sheet and has a high degree of confidence and an appropriate class probability. In addition, the processor 210 may store the sheet detection model NN2 in the storage device 215 in S580, or alternatively, may store it in an external storage device (not shown) connected to the information processing device 200.

[0107] The trained sheet detection model NN2 (FIG. 8(A)) can be used to inspect a printer. Processor 210 inputs captured image data of the printer (for example, the captured images described in FIGS. 2(A) and 2(B)) to sheet detection model NN2. Output data 730 output from sheet detection model NN2 indicates the area of ​​the label sheet detected from the captured image. If a label sheet is detected, the printer inspection result is a pass. If a label sheet is not detected, the printer inspection result is a fail.

[0108] As described above, in the process of FIG. 9, the processor 210 executes a process of associating information with the second type of learning image data D21 used for training the sheet detection model NN2, which is an example of a machine learning model. Specifically, in S310, the processor 210 acquires target image data (i.e., the second type of learning image data D21) of a target image (e.g., the target image 700a (FIG. 10(A))) which is an image to be processed. In S315, the processor 210 detects a logo area (e.g., the logo area LA) indicating an image of a logo, which is an example of a target object of interest, from within the target image by analyzing the target image data. In S325 and S340, the processor 210 determines a candidate extension area LAe including the logo area and a part outside the logo area. In S360 and S365, the processor 210 associates annotation data indicating annotation information including area information indicating the extension area LAe (i.e., the label data D22) with the target image data and stores it in the storage device 215. In this way, the processor 210 can associate region information indicating the candidate extended region LAe that includes other regions in addition to the region LA showing the logo with the second type of learning image data D21. For example, as described in Figures 12(A) and 12(B), the processor 210 can associate region information D221 indicating the region of the label sheet 910L that includes the logo image 910 and other images with the second type of learning image data D21. Such region information D221 is suitable for training a machine learning model (e.g., sheet detection model NN2) that processes regions that include other regions in addition to the region showing the logo (e.g., the region of a label sheet).

[0109] As described in S310 (FIG. 9), FIG. 10(A), etc., the image of the target object is a logo image. Therefore, the processor 210 can associate area information D221 indicating an extended area including other areas in addition to the area indicating the logo image with the second type learning image data D21.

[0110] Also, as shown in FIG. 10(B) and the like, the area detected in S315 (FIG. 9) is a rectangular area. In order to detect a rectangular area showing an image of an object of interest (e.g., a logo image), various object detection models can be used, not limited to YOLO (e.g., SSD (Single Shot MultiBox Detector), R-CNN (Region Based Convolutional Neural Networks), etc.). Therefore, the processor 210 can appropriately detect the area.

[0111] The process of determining the extension area also includes S325 (FIG. 9). S325, i.e., the process of FIG. 11, includes a process of extending the extension area from the same area as the logo area LA toward the outside of the logo area LA by analyzing the target image data, as described with reference to FIG. 10(E) etc. In this embodiment, this process is performed by the processor 210. Furthermore, this process determines candidates for the extension area LAe. In this way, since the processor 210 (i.e., the information processing device 200) extends the extension area LAe, the information processing device 200 can reduce the burden on the user.

[0112] Also, S325, i.e., the process of FIG. 11, includes the processes of S410-S430 and S440. As described in FIG. 10(D) and the like, in the processes of S410-S430, the processor 210 analyzes the target image data to identify a block BL having an edge intensity value equal to or less than a reference value as a uniform block BL1. The edge intensity value is an evaluation value of the ratio of a change in color to a change in position on the target image. The condition (also called the uniform condition) for selecting a block BL as a uniform block BL1 indicates that the edge intensity value is equal to or less than a reference value. Also, as described in FIG. 10(E) and the like, in S440, the processor 210 extends the extension region toward the outside of the logo region LA so that the entire contour LAeo of the extension region LAe is included in the uniform block BL1. In this way, the processor 210 can appropriately extend the extension region LAe by using the uniform block BL1. For example, the extended region LAe can be extended to the boundary between the region of a large object (e.g., a label sheet 910L) including the target object (here, the logo image 910) and other elements and the background region. The extended region LAe is suitable for training a machine learning model (e.g., a sheet detection model NN2) that processes the region of a large object including other regions in addition to the region showing the logo.

[0113] Moreover, the process of determining the extension area LAe includes S335 and S340 (FIG. 9). In S335, as described with reference to FIGS. 12(A), 12(B), etc., the processor 210 displays on the display unit 240 a first user interface image 610 for allowing the user to change the position of the contour LAeo of the candidate extension area LAe. In S340, the processor 210 determines, as the extension area, an area having a contour whose position has been changed by the user. Thus, the processor 210 can determine an appropriate extension area by using the contour changed by the user.

[0114] Also, when S355 (FIG. 9) is executed, in S335, as explained in FIG. 12(A) etc., the processor 210 displays on the display unit 240 a second user interface image 620 for allowing the user to specify sheet class information indicating the classification of the extended area LAe (i.e., the classification of the label sheet). In S360 and S365, the processor 210 associates annotation data indicating annotation information including the sheet class information specified by the user with the target image data and stores it in the storage device 215. Thus, the processor 210 can associate appropriate sheet class information with the target image data.

[0115] 12(A), 12(C), etc., the second user interface image 620 includes a candidate area 621 that shows one or more candidates of sheet class information selectable by the user. As described in S330, the candidate area 621 shows, as one or more candidates, one or more pieces of sheet class information that are pre-associated with the logo included in the logo area detected in S315, among the C pieces of pre-determined sheet class information. For example, when the first logo image 910 (FIG. 12(A)) is detected, the sheet class information CC1 and CC2 associated with the first logo image 910 are the candidates. This allows the user to easily select appropriate sheet class information.

[0116] Furthermore, when S350 (FIG. 9) is executed, in S350, the processor 210 determines candidate sheet class information, which is previously associated with the logo included in the logo area detected in S315, among the predetermined C pieces of sheet class information, as sheet class information to be included in the annotation information. For example, when the third logo image 930 (FIG. 12(C)) is detected, in S330, the processor 210 selects the third sheet class information CC3 associated with the third logo image 930 as a candidate. In S350, the processor 210 determines the third sheet class information CC3 as the sheet class information to be included in the annotation information. Then, in S360 and S365, the processor 210 associates annotation data indicating annotation information including the determined sheet class information with the target image data and stores it in the storage device 215. Thus, the processor 210 can associate appropriate sheet class information with the target image data.

[0117] B. Variations: (1) The process of generating learning image data for training an object detection model may be various other processes instead of the process in Fig. 4. For example, the process of dividing the logo image into T types of partial regions (S115-S120) may be a process of dividing the logo image according to a predetermined region pattern (for example, a region pattern indicating a first type region A1 and a second type region A2) without analyzing the color distribution of the logo image data.

[0118] In S125, the changed color may be various colors. For example, the changed color may be a predetermined color different from the original color. In addition, if the color of the logo image is expressed using halftone dots, the changed color may be a color expressed using halftone dots different from the original halftone dots (for example, a different number of lines from the original number of lines).

[0119] The background images available in S135-S140 are not limited to single-color solid images and photographs, but may be various images such as figures, patterns, etc. Also, from the available background images, either one of the photograph and the single-color solid image, or both, may be omitted.

[0120] In S130, either one of the size change process and the aspect ratio change process may be omitted, or S130 may be omitted.

[0121] In each of one or more of the processes of S125, S130, S135, S140, and S145, the processor 210 may determine the process content according to a predetermined plan, rather than randomly determining the process content.

[0122] The number C of logo image types (ie, the number C of classifications (classes)) is not limited to 3, but may be any number greater than 1, such as 1, 2, 4, or the like.

[0123] In S140, multiple logo images may be arranged so that they partially overlap each other, or parts of the logo images may be deleted.

[0124] In S145, one or more processes arbitrarily selected from the above seven processes P1 to P7 may be omitted from the available processes. Also, S145 may be omitted.

[0125] The machine learning model for detecting logo images is not limited to the YOLO model described in Fig. 3(A) and may be an improved YOLO model such as "YOLO v3". In addition, other models such as SSD, R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN may be used.

[0126] The object to be detected by the object detection model is not limited to an image of a logo, but may be any object (for example, a part to be attached to a printer, a barcode, etc.). The first type data set DS1 generated by the process of Fig. 4 (or the process of a modified example thereof) may be used for training various object detection models.

[0127] (2) The process of associating information with image data used in training the machine learning model may be various other processes instead of the process of Fig. 9. For example, the process of detecting a logo area (S315) may be various other processes instead of the process using the logo detection model NN1. For example, the processor 210 may detect the logo area by pattern matching using reference logo image data indicating a reference logo image.

[0128] The process of determining the extension region may be various other processes instead of the processes of S325 and S340. For example, the processor 210 may determine the extension region using one template image showing a logo region and an extension region associated with the logo region. Specifically, the processor 210 determines the position of the template image with respect to the target image so that the logo region in the target image matches the logo region of the template image. Then, the processor 210 determines the extension region shown by the template image at the determined position as the extension region to be applied to the target image.

[0129] The target object used to determine the extended area is not limited to a logo image, and may be any object, such as a barcode. The shape of the target object area (e.g., logo area) showing the image of the target object may be any shape other than a rectangle. For example, the shape of the target object area may be a polygon such as a triangle, pentagon, or hexagon, or may be a shape determined by a contour including curved portions, such as a circle or ellipse. The shape of the target object area may be a shape determined by the contour of the target.

[0130] The process of identifying a uniform region on the target image may be various other processes instead of the processes of S410-S430 in FIG. 11. Here, the uniform region is a region that satisfies a uniform condition. The uniform condition is a condition indicating that the rate of color change with respect to a change in position on the target image is equal to or lower than a reference. For example, the edge strength value of the block BL may be various values ​​indicating the rate of color change with respect to a change in position. The edge strength value may be, for example, the difference between the maximum luminance value and the minimum luminance value in the block BL. The processor 210 may also identify the uniform region using a histogram of color values ​​(e.g., luminance values) of multiple pixels of the target image. Specifically, the processor 210 may identify a continuous region formed by multiple pixels included in one section of the histogram as a uniform region. In this case, the uniform condition is that the color value is included in one section.

[0131] 12(A)-12(C), one UI screen 600 shows a first user interface image 610 and a second user interface image 620. That is, the process of displaying the UI screen 600 on the display unit 240 includes a process of displaying the first user interface image 610 and a process of displaying the second user interface image 620. Alternatively, the processor 210 may display the first user interface image 610 on a screen different from the screen showing the second user interface image 620.

[0132] 9 may be omitted. In this case, the processor 210 may determine the candidate extension region determined in S325 as the final extension region as is. Also, the first user interface image 610 may be omitted from the UI screen (FIGS. 12(A)-12(C)).

[0133] S350 in FIG. 9 may be omitted. For example, the processor 210 may select multiple candidate sheet class information in S330 regardless of the logo class specified in S315, and may accept the sheet class information specified by the user in S355. S355 may also be omitted. For example, the processor 210 may select one candidate sheet class information associated with the logo class in S330 regardless of the logo class specified in S315, and may determine the sheet class information to be the candidate sheet class specified in S330 in S350. The sheet class information may be omitted from the annotation information. For example, when the number C of types of label sheets is 1, appropriate training using the second type data set DS2 is possible even if the sheet class information is omitted.

[0134] The machine learning model for detecting the image of the label sheet is not limited to the YOLO model described in FIG. 8(A), but may be other models such as YOLO v3, SSD, R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN.

[0135] (3) The inspection target using the machine learning model is not limited to a printer, and may be any product such as a scanner, a multifunction printer, a digital camera, a cutting machine, or a mobile terminal. Also, a case that contains a product may be the inspection target. Also, the machine learning model may be trained to detect various other objects, not limited to a label sheet. For example, the machine learning model may detect parts to be attached to a printer from a captured image of the printer. In either case, when an image of a detection target (e.g., a label sheet) includes an image of a small characteristic part (e.g., a logo), an extended region including the characteristic part can be used as a region indicating the detection target. Annotation information including region information indicating such an extended region may be associated with image data for training. Such image data and annotation information may be used for training various machine learning models, such as classification models, not limited to object detection models.

[0136] (4) The color space of the input image data input to the machine learning model may be another color space such as the CMYK color space instead of RGB. The input image data may represent an image by a luminance value. The input image data may be generated by performing various image processes such as a resolution conversion process and a cropping process.

[0137] (5) The method of associating image data with label data may be any method. For example, the label data may include identification data that identifies the image data associated with the label data. The processor 210 may generate table data that indicates the correspondence between the image data and the label data. The processor 210 may store the image data and label data associated with each other in one data file.

[0138] (6) The training process of the machine learning model may be various methods suitable for the machine learning model instead of the processes of Figures 7 and 13. For example, in the embodiments of Figures 7 and 13, the loss function used to calculate the loss may be various functions that calculate an evaluation value of the difference between the output data 730, 830 and the label data, such as a cross-entropy error. For example, when an object detection model is used, the loss function may be various functions that calculate a loss that is correlated with the error of the area showing the object and the error of the probability of each type of object.

[0139] In addition, the method of adjusting the calculation parameters included in the machine learning model may be various other methods such as a method of propagating a target value (also called Target Propagation) instead of the error backpropagation method. In addition, the condition for completing the training may be various conditions that indicate that the difference between the data output from the machine learning model and the label data is small. For example, the confirmation loss may be omitted from the condition for completing the training. In this case, in the processing of FIG. 7 and FIG. 13, all the labeled data may be used as the learning data set. In addition, the processor 210 may determine that the training is completed when an instruction to complete the training is input from the operator, and may determine that the training is not completed when an instruction to continue the training is input. For example, the operator may determine whether to end the training by referring to the output data output using the confirmation data set. Alternatively, the condition for completing the training may be that the calculation of the learning loss and the update of the calculation parameters (for example, S240-S250 (FIG. 7), S540-S550 (FIG. 13)) have been repeated a predetermined number of times.

[0140] (7) The data set generation process of Fig. 4, the training process of Fig. 7, the annotation process (data set generation process) of Fig. 9, the training process of Fig. 13, and an inspection process (not shown) may be executed by different information processing devices. Any process selected from these processes may be shared among a plurality of devices (e.g., information processing devices such as computers) that can communicate with each other via a network.

[0141] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the functions of the models NN1 and NN2 in FIG. 1 may be realized by dedicated hardware circuits.

[0142] Furthermore, when some or all of the functions of the present invention are realized by a computer program, the program can be provided in a form stored in a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used in a state stored in the same or a different recording medium (computer-readable recording medium) from when it was provided. "Computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0143] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the invention, and equivalents thereof are included in the present invention. [Explanation of symbols]

[0144] 100...digital camera, 200...information processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...first program, 232...second program, 233...third program, 234...fourth program, 240...display unit, 250...operation unit, 270...communication interface, 610...first user interface image, 620...second user interface image, 621...candidate area, 622...check box, DV...object (printer), DVx...first printer, DVy...second printer, Dx...first direction, Dy...second direction

Claims

1. 1. A method of associating information with image data used to train a machine learning model, comprising: An acquisition step of acquiring target image data of a target image which is an image to be processed; a detection step of detecting an object region in the object image, which represents an image of an object of interest, by analyzing the object image data; determining an extension region including the object region and a portion outside the object region; an associating step of associating annotation data indicating annotation information including area information indicating the extended area with the target image data and storing the annotation data in a storage device; The method of association comprises:

2. The method of claim 1 , further comprising: The image of the object of interest is a logo image. Association method.

3. 3. The method of claim 1, further comprising: The object region is a rectangular region. Association method.

4. 4. The method of claim 1, further comprising: The determination step includes a device expansion step in which the information processing device expands the expansion region from the same region as the object region toward the outside of the object region by analyzing the object image data. Association method.

5. The method of claim 4, further comprising the steps of: The device expansion step includes: A step of identifying a uniform region, which is a region that satisfies a uniformity condition indicating that a rate of color change relative to a position change on the target image is equal to or less than a reference value, by analyzing the target image data by the information processing device; a step of expanding the extension region toward an outside of the object region such that the entire contour of the extension region is included in the uniform region by the information processing device; A method of associating,

6. 6. The method of claim 1, further comprising: The determining step includes: displaying on a display device a first user interface image for allowing a user to change the position of the contour of the extended region; determining an area having an outline of the user-modified position as the extended area; Including, Association method.

7. The method of claim 1 , further comprising: displaying, on a display device, a second user interface image for allowing a user to specify class information indicating a classification of the extension region; The associating step associates the annotation data indicating the annotation information including the class information specified by the user with the target image data and stores the annotation data in the storage device. Association method.

8. The method of claim 7, further comprising the steps of: the second user interface image includes a candidate area showing one or more candidates of class information selectable by the user; The candidate region indicates, as the one or more candidates, one or more pieces of class information that are preliminarily associated with the target object included in the detected target object region, among a plurality of pieces of preliminarily determined class information. Association method.

9. 7. The method of claim 1, further comprising: In the associating step, the information processing device associates the annotation data indicating the annotation information including class information, which is previously associated with the target object included in the detected target object region, with the target image data and stores the annotation data in the storage device. Association method.

10. 1. A system for associating information with image data used to train a machine learning model, comprising: an acquisition unit that acquires target image data of a target image that is an image to be processed; a detection unit that detects an object region indicative of an image of an object of interest from within the object image by analyzing the object image data; a determination unit for determining an extension region including the object region and a portion outside the object region; an associating unit that associates annotation data indicating annotation information including area information indicating the extended area with the target image data and stores the annotation data in a storage device; A system comprising:

11. 1. A computer program for a computer that associates information with image data used to train a machine learning model, comprising: An acquisition function for acquiring target image data of a target image which is an image to be processed; a detection function for detecting an object region in the object image, the object region representing an image of an object of interest, by analyzing the object image data; a determining function for determining an extension region including the object region and a portion outside the object region; an associating function for associating annotation data indicating annotation information including area information indicating the extended area with the target image data and storing the annotation data in a storage device; A computer program that enables a computer to realize the above.

Images