Computer program, generating device, and generating method

Abstract

To appropriately generate data for inspection.SOLUTION: A generation device detects, from first photographic image data representing a first photographic image of a first type physical object including K (K is an integer of 1 or more) objects and having no abnormality in its appearance, K object areas corresponding to the K objects by using a trained object detection model. The generation device generates first correspondence data indicating K types of correspondence corresponding respectively to the K object areas. The K types of correspondence each indicate the correspondence between object area information being information for defining the object areas in the photographic image of the first type physical object and condition information indicating inspection conditions. The condition information indicates, of L (L is an integer of 1 or more and K or less) inspection conditions, inspection conditions associated with the types of the object areas. The generation device stores the first correspondence data in a storage.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present specification relates to data for inspecting the appearance of an object. [Background technology]

[0002] Conventionally, captured images have been used for various processes. For example, Patent Document 1 proposes a technology for performing object recognition for autonomous driving using captured images of the area in front of a vehicle. The captured image may include multiple image regions with different resolutions. Object recognition processing is performed based on different criteria for image regions with low resolution and other image regions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2017 / 163606 Summary of the Invention [Problem to be solved by the invention]

[0004] The captured image can be used to inspect the appearance of various objects, such as label sheets installed in a printer. The object can include one or more objects (e.g., character strings, marks, etc.). The appearance of each object can be inspected for abnormalities. Here, the object can be located at various positions within the captured image. Also, appropriate inspection conditions can vary depending on the object. In the inspection, data indicating the correspondence between information (e.g., coordinates) defining the area of ​​the object within the captured image and the inspection conditions can be referenced. Such data is generated, for example, by an operator. Generating such data is a significant burden for the operator.

[0005] This specification discloses a technique for appropriately generating data for inspection. [Means for solving the problem]

[0006] The techniques disclosed in this specification can be implemented in the following application examples.

[0007] [Application Example 1] A computer program for a computer that generates data for inspecting the appearance of an object, the computer program causing the computer to realize: a first area detection function that uses a trained object detection model to detect K object areas corresponding to K objects (K is an integer greater than or equal to 1) from first captured image data that represents a first captured image of a first type of object that includes the K objects and has no abnormalities in appearance; and a first data generation function that generates first correspondence data that indicates K correspondences corresponding to the K object areas, each of the K correspondences indicating a correspondence between object area information that is information that defines an object area in the captured image of the first type of object, and condition information that indicates an inspection condition, and the condition information indicates an inspection condition that is associated with the type of object area out of L inspection conditions (L is an integer greater than or equal to 1 and less than or equal to K).

[0008] According to this configuration, the first correspondence data that can be used to determine the inspection conditions for the object region when inspecting the first type object is generated appropriately.

[0009] The technology disclosed in this specification can be realized in various forms, such as a method and device for generating inspection data, an inspection method and device for using inspection data, a computer program for realizing the functions of those methods or devices, a recording medium (e.g., a non-temporary recording medium) on which that computer program is recorded, and the like. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is an explanatory diagram illustrating a data processing system according to an embodiment; [Figure 2]1 is a perspective view of a digital camera 110, a multifunction peripheral 900, and a support base 190. FIG. [Figure 3] 1A is a schematic diagram showing an example of a label sheet, FIG. 1B is a schematic diagram showing an example of a captured image, and FIG. 1C is a schematic diagram showing an example of an object region. [Figure 4] 10 is a flowchart illustrating an example of a generation process. [Figure 5] FIG. 10 is a schematic diagram showing an example of correspondence data D1. [Figure 6] 10 is a flowchart illustrating an example of an object type determination process. [Figure 7] 10 is a flowchart illustrating an example of an inspection process. [Figure 8] 10 is a flowchart illustrating another embodiment of the object type determination process. [Figure 9] 10 is a flowchart illustrating another embodiment of the inspection process. [Figure 10] 10 is a flowchart illustrating another embodiment of the object type determination process. [Figure 11] FIG. 1 is a schematic diagram illustrating an example of a classification model. [Figure 12] 10 is a flowchart showing another embodiment of the process for generating correspondence data D1. [Figure 13] FIG. 10 is a schematic diagram showing another example of a label sheet. [Figure 14] 10 is a flowchart illustrating an example of a generation process. DETAILED DESCRIPTION OF THE INVENTION

[0011] A. First Example: A1.Device configuration: FIG. 1 is an explanatory diagram showing a data processing system according to one embodiment. In this embodiment, a data processing system 1000 includes a first data processing device 200 and a second data processing device 300. The data processing devices 200 and 300 are, for example, personal computers. The first data processing device 200 is an example of a generating device that generates data used in inspecting the appearance of an object (for example, a label sheet attached to a product such as a multifunction peripheral). The second data processing device 300 is an example of an inspecting device that inspects the appearance of an object. Hereinafter, it is assumed that the appearance of a label sheet 800 attached to a multifunction peripheral 900 is to be inspected.

[0012] The first data processing device 200 includes a processor 210, a storage device 215, a display unit 240, an operation unit 250, and a communication interface 270. These elements are connected to each other via a bus. The storage device 215 includes a volatile storage device 220 and a non-volatile storage device 230.

[0013] The processor 210 is a device configured to perform data processing, and is, for example, a CPU. The volatile storage device 220 is, for example, a DRAM, and the non-volatile storage device 230 is, for example, a flash memory. The non-volatile storage device 230 stores a generation program 231, an object detection model M1, a character recognition module M2, and correspondence data D1. In this embodiment, the object detection model M1 and the character recognition module M2 are each program modules. The object detection model M1 is a so-called machine learning model. The processor 210 generates the correspondence data D1 in accordance with the generation program 231. Details of the generation program 231, the object detection model M1, the character recognition module M2, and the correspondence data D1 will be described later.

[0014] The display unit 240 is a device configured to display images, such as a liquid crystal display or an organic EL display. The operation unit 250 is a device configured to receive user operations, such as buttons, levers, or a touch panel overlaid on the display unit 240. By operating the operation unit 250, the user can input various requests and instructions to the first data processing device 200. The communication interface 270 is an interface for communicating with other devices (for example, a USB interface, a wired LAN interface, or an IEEE802.11 wireless interface). A digital camera 110 is connected to the communication interface 270. The digital camera 110 is used to photograph the label sheet 800 of the multifunction peripheral 900.

[0015] The hardware configuration of the second data processing device 300 is the same as that of the first data processing device 200. The second data processing device 300 includes elements 310, 315, 320, 330, 340, 350, and 370 corresponding to the elements 210, 215, 220, 230, 240, 250, and 270 of the first data processing device 200, respectively. A digital camera 110 is connected to a communication interface 370. A non-volatile storage device 330 of the second data processing device 300 stores an inspection program 331, correspondence data D1, and reference image data D2. The correspondence data D1 is generated by the first data processing device 200 and copied from the first data processing device 200 to the second data processing device 300. The reference image data D2 is used in the inspection process described below. Details of the inspection program 331, the correspondence data D1, and the reference image data D2 will be described later.

[0016] 2 is a perspective view of the digital camera 110, the multifunction device 900, and the support base 190. The support base 190 supports the multifunction device 900. In this embodiment, the support base 190 forms a flat upper surface 191. A bottom surface 909 of the multifunction device 900 is placed on the upper surface 191. A label sheet 800 is affixed to a first side surface 901 of the multifunction device 900. The digital camera 110 is positioned so as to photograph the label sheet 800.

[0017] A2.Label sheet: FIG. 3A is a schematic diagram illustrating an example of a label sheet. FIG. 3A shows a label sheet 800 without any defects. In this embodiment, the label sheet 800 includes a logotype 810, a certification mark 820, a description 830, a trademark 840, a first character string 850, a photograph 860, and a second character string 870. Hereinafter, the total number of objects included in the label sheet 800 is defined as K (in this embodiment, K=7). Details of the certification mark 820, the description 830, the trademark 840, and the photograph 860 are omitted from the figure. The certification mark 820 is a mark established based on a standard or law, such as the CE mark, GS mark, or FCC mark. Such marks indicate compliance with a standard or law. Compliance with a specific standard may be required by law. Such marks indicating compliance are also a type of mark established based on a standard or law. The description 830, for example, provides notes based on the standard or law. The trademark 840 represents, for example, a mark indicating the manufacturer of the multifunction device 900. The first character string 850 represents a model number. The photo 860 represents, for example, a user operating the multifunction device 900. The second character string 870 represents the country of manufacture and includes the character string "MADE IN." In this way, the label sheet 800 includes various types of objects.

[0018] In this embodiment, the label sheet 800 is inspected using a captured image of the label sheet 800 taken by the digital camera 110 (FIG. 1). FIG. 3B is a schematic diagram showing an example of a captured image. The captured image 700 is a rectangular image with two sides parallel to the first direction Dx and two sides parallel to the second direction Dy perpendicular to the first direction Dx. The captured image 700 is represented by the color values ​​of multiple pixels arranged in a matrix along the first direction Dx and the second direction Dy. In this embodiment, the color values ​​are represented by three component values: R (red), G (green), and B (blue), and each component value is represented in 256 levels, for example, from 0 to 255. The captured image 700 in the figure represents a label sheet 800 without any defects.

[0019] In this embodiment, the label sheet 800 is inspected to determine whether the appearance of each of the objects 810-870 is normal. The objects may have various defects, such as chips, deformations, or stains. If the defects are minor, the appearance of the object is determined to be normal. As described below, the criteria for determining whether the appearance of an object is normal may vary depending on the type of object.

[0020] For character images that represent a relatively small number of characters, such as the logotype 810, the first character string 850, and the second character string 870, if the defect is small, the user can read the correct information from the entire character image. For example, even if some characters are difficult to read, the user can read the correct information from the entire character image. Such small defects may be acceptable.

[0021] A text image that displays a relatively large number of characters, such as the explanatory text 830, may represent important information. In this case, it is preferable that the text image clearly displays the correct information. Therefore, it is preferable that the number of defects that can be tolerated is smaller than that permitted for a text image that is composed of a relatively small number of characters. For example, it is preferable that defects that make some characters difficult to read are not tolerated.

[0022] For a simple illustration such as the trademark 840, if the defect is small, the user can read the correct information from the entire image. For example, even if a part of the illustration has a defect, the user can read the correct information from the entire illustration. Such a small defect may be acceptable.

[0023] With respect to the authentication mark 820, it is preferred that the label sheet 800 represent the correct mark. Any imperfections that cause the shape of the mark to differ from the correct shape are preferably not allowed.

[0024] For images that display complex shapes and many colors, such as photographs 860 and complex illustrations, defects are less noticeable, and therefore, larger defects can be tolerated than for other types of objects.

[0025] Thus, the preferred inspection criteria may vary depending on the type of object area. The correspondence between the object area and the object area type (or more generally, the inspection criteria) can be determined by the operator. However, when multiple objects 810-870 are included in the label sheet 800, the burden of determining the correspondence is significant. In this embodiment, the first data processing device 200 (FIG. 1) uses the captured image data to generate correspondence data D1 indicating the correspondence.

[0026] A3.Generation process: FIG. 4 is a flowchart showing an example of the generation process. The data processing device 200 (FIG. 1) generates correspondence data D1 by executing the generation process. FIG. 5 is a schematic diagram showing an example of the correspondence data D1. The correspondence data D1 indicates the correspondence between object numbers D1a, object region information D1b, and object types D1c. The object numbers D1a are the identification numbers of the multiple objects 810-870 included in the label sheet 800 (FIG. 3(A)). The object region information D1b is information that defines an object region, which is a region that represents an object in a captured image (for example, the captured image 700 (FIG. 3(B)). In this embodiment, the object region is a rectangular region surrounded by two sides parallel to the first direction Dx and two sides parallel to the second direction Dy. The object region information D1b indicates a combination of coordinates D1b1 of the upper left corner of the rectangle and coordinates D1b2 of the lower right corner of the rectangle. Each of the coordinates D1b1, D1b2 indicates the position of a pixel. The object type D1c indicates an identification number for the type of object region. As will be described later, the object type D1c indicates inspection conditions (the object type D1c will also be referred to as condition information D1c).

[0027] The worker arranges the multifunction device 900 and the digital camera 110 as shown in Figure 2. A label sheet 800 without any defects is attached to the multifunction device 900. The worker connects the digital camera 110 to the communication interface 270 of the first data processing device 200 (Figure 1). The worker then operates the operation unit 250 to input an instruction to start the generation process. In response to the instruction, the processor 210 executes the generation process of Figure 4 in accordance with the generation program 231.

[0028] In S110, the processor 210 issues a capture instruction to the digital camera 110. In response to the instruction, the digital camera 110 captures the label sheet 800 and generates image data representing the captured image. In S120, the processor 210 acquires image data from the digital camera 110. Hereinafter, image data representing the captured image will be referred to as captured image data. In this embodiment, the image size of the captured image data used in the generation process and the inspection process described below (specifically, the number of pixels in the first direction Dx (FIG. 3B) and the second direction Dy) is predetermined (referred to as the processed image size). The processor 210 acquires captured image data of the processed image size representing the label sheet 800 by performing various image processing on the image data acquired from the digital camera 110, such as cropping to cut out the portion representing the label sheet 800 and resolution conversion. It is preferable to adjust the positioning of the multifunction peripheral 900 and the digital camera 110 so that the area of ​​the label sheet 800 is extracted by cropping the predetermined area. Alternatively, processor 210 may detect the area of ​​label sheet 800 (e.g., pattern matching) and perform a trimming process to extract the detected area of ​​label sheet 800. Hereinafter, captured image data will be assumed to represent captured image 700 in Figure 3(B).

[0029] In S130, the processor 210 uses the trained object detection model M1 to detect object regions of each of K objects 810-870 (in this embodiment, K=7) from the captured image. Various object detection models can be used as the object detection model M1. In this embodiment, the object detection model M1 is an object detection model called YOLO (You Only Look Once). YOLO is disclosed, for example, in the paper "Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788."

[0030] The YOLO model uses a convolutional neural network to predict a rectangular frame containing an object, called a bounding box, the confidence that the box contains the object, and the probability of each type of object (also called class probability) if the box contains the object. If the area enclosed by the box represents the object to be detected, a high confidence is calculated. If the area enclosed by the box does not represent the object to be detected, a low confidence is calculated. A box with a confidence equal to or greater than a threshold is used as a box representing the detected object. A box with a confidence below the threshold is treated as a box that does not represent the detected object. The confidence threshold is predetermined. Alternatively, the confidence threshold may be adjusted based on the results of training the object detection model M1. The highest class probability among the multiple class probabilities associated with the box indicates the type of object enclosed by the box. The output data from the object detection model M1 represents multiple combinations of bounding boxes, confidences, and class probabilities based on the image data input to the object detection model M1.

[0031] 3(C) is a schematic diagram showing an example of an object region detected by the object detection model M1. The figure shows an example in which image data of the captured image 700 (FIG. 3(B)) is input to the object detection model M1. As shown in the figure, the object detection model M1 detects bounding boxes BB1-BB7 that surround each of the objects 810-870. In this example, the object detection model M1 is pre-trained to detect the regions of each of the seven types of objects 810-870 included in the label sheet 800.

[0032] Class identifiers CL1-CL7 are associated with the bounding boxes BB1-BB7, respectively. The first class identifier CL1 indicates the type associated with the highest class probability of the first bounding box BB1. In this embodiment, the first class identifier CL1 indicates a logotype 810. Similarly, the class identifiers CL2-CL7 of the other bounding boxes BB2-BB7 indicate the types associated with the highest class probabilities. In this embodiment, the class identifiers CL2-CL7 indicate objects 820-870, respectively. In this manner, the object detection model M1 detects seven types of objects 810-870.

[0033] The object detection model M1 is trained using image data of multiple images including images of objects 810-870. A training processor (e.g., processor 210) uses the image data to perform operations on the object detection model M1, thereby generating output data. The training processor then adjusts multiple operation parameters of the object detection model M1 so that the output data approaches the correct data. For example, image data of an image including an image of the object 810 is input to the object detection model M1. In this case, the multiple operation parameters of the object detection model M1 are adjusted so that the output data indicates a bounding box surrounding the object 810, a high confidence associated with the bounding box, and a high class probability associated with the bounding box that indicates the object 810. Similarly, the multiple operation parameters of the object detection model M1 are adjusted for the other objects 820-870.

[0034] Various methods may be used to adjust the calculation parameters. In this embodiment, multiple calculation parameters of the object detection model M1 are adjusted so that the loss calculated using a loss function is reduced. The loss function may be any of various functions that calculate an evaluation value of the difference between output data and ground truth data associated with image data input to the object detection model M1. In this embodiment, the loss function disclosed in the above-mentioned paper by YOLO is used. Furthermore, as an algorithm for adjusting the multiple calculation parameters, for example, an algorithm using backpropagation and gradient descent may be adopted. Here, so-called Adam optimization may be performed. Note that the ground truth data is generated in advance by an operator.

[0035] In S130 (FIG. 4), the processor 210 generates output data by executing the operations of the trained object detection model M1 (FIG. 1) using the image data of the captured image 700. The generated output data indicates the bounding boxes BB1-BB7 described in FIG. 3(C).

[0036] In S135, the processor 210 acquires object area image data representing an image of the object area from the captured image data for each of the K object areas (in this embodiment, K=7). The processor 210 stores the acquired data in the storage device 215 (for example, the non-volatile storage device 230).

[0037] In S140, the processor 210 initializes the correspondence data D1 (FIG. 5). In this embodiment, the processor 210 generates empty correspondence data D1.

[0038] In S150, the processor 210 selects an unprocessed object region from the K object regions detected in S130 as an object region of interest.

[0039] In S160, the processor 210 executes an object type determination process. This process is a process for determining the type of the target object region. Fig. 6 is a flowchart showing an example of the object type determination process.

[0040] In S210, the processor 210 generates a histogram for each color component using the object region image data of the target object region. In this embodiment, three histograms for the three color components of RGB are generated.

[0041] In S220, the processor 210 calculates a peak value of each color component using the histogram. The peak value is the component value that shows the highest frequency. Note that the background color range may be excluded from the calculation of the peak value. The background color range may be, for example, a color range corresponding to black (a color range including zero) or a color range corresponding to white (a color range including 255). The background color range may be determined in advance. Alternatively, the processor 210 may determine a background color value by analyzing the captured image, and then determine a background color range that includes the determined background color value.

[0042] In S230, processor 210 determines whether the peak value is less than the peak threshold. The peak value may vary depending on the type of object region. For example, if the attention area image represents a large number of colors, such as a photograph or a complex illustration, each component value will be distributed over a wide range from small to large values. As a result, the peak value may be small. If the attention area image represents a small number of colors, such as a simple illustration such as a trademark, an authentication mark, or a character string, the peak value of each color component typically indicates the color of the object. The color of the object may be a color represented using large component values, such as red, green, blue, or yellow. In this case, the peak value of one or more color components may be large. In this way, the peak value can be used to classify the type of object region.

[0043] In this embodiment, processor 210 determines whether a first condition is satisfied, that is, all peak values ​​of all color components are less than a peak threshold. If the first condition is satisfied (S230: Yes), processor 210 determines in S260 that the type of the target object region is a second image type, and ends the processing of FIG. 6. The second image type is a type that includes photographs and complex illustrations. The peak threshold is experimentally determined in advance so that the second image type, which includes photographs and complex illustrations, can be classified from other types.

[0044] If the first condition is not satisfied, i.e., if the peak values ​​of one or more color components are equal to or greater than the peak threshold (S230: No), in S240, the processor 210 executes character recognition processing of the attention area image in accordance with the character recognition module M2. The character recognition processing may be any of various known processes. For example, an optical character recognition engine called "Tesseract OCR" by Google Inc. may be used. Alternatively, pattern matching using pre-prepared character images may be used.

[0045] In S250, processor 210 determines whether or not characters were recognized from the attention area image in S240. If characters are recognized (S250: Yes), in S280 processor 210 determines that the type of attention object area is a character type, and ends the processing of Fig. 6. The character type is a type of object area that represents characters, and is a type that includes logotype, sentence, and character.

[0046] If no characters are recognized in S240 (S250: No), in S270, processor 210 determines that the type of the target object region is the first image type, and ends the processing of Fig. 6. The first image type is a type that includes marks established based on laws or standards (certification marks, statutory marks, standard marks, etc.) and simple illustrations such as icon images and trademarks.

[0047] As described above, in the object type determination process of Fig. 6, processor 210 determines the type of the attention object area by analyzing the attention area image based on predetermined rules. Processor 210 determines the type of the attention object area to be one of three types: a first image type, a second image type, and a character type.

[0048] After the object type determination process, i.e., S160 in FIG. 4, in S170, processor 210 adds data indicating the correspondence of the target object regions to correspondence data D1 (FIG. 5). Object numbers D1a are assigned in ascending order starting from 1. Object region information D1b indicates the coordinates of the corners of the bounding box indicating the target object region. Object type D1c indicates the type of object region determined in S160 (for example, 1 indicates the first image type, 2 indicates the second image type, and 3 indicates the character type).

[0049] In S180, processor 210 determines whether all object regions have been processed. If unprocessed object regions remain (S180: No), processor 210 proceeds to S150 to process the unprocessed object regions. If processing of all object regions has been completed (S180: Yes), in S190 processor 210 stores correspondence data D1 in storage device 215 (e.g., non-volatile storage device 230) and terminates the processing of Figure 4. The correspondence data D1 generated by the above generation process indicates the correspondences between all objects 810-870 on label sheet 800 (Figure 3(A)).

[0050] A4. Inspection process: FIG. 7 is a flowchart illustrating an example of the inspection process. The second data processing device 300 (FIG. 1) executes the inspection process to inspect the label sheet 800 of the multifunction peripheral 900 (FIG. 2). In the inspection process, the processor 310 (FIG. 1) uses the correspondence data D1 and the reference image data D2. The operator copies the correspondence data D1 generated in the process of FIG. 4 from the nonvolatile storage device 230 of the first data processing device 200 to the nonvolatile storage device 330 of the second data processing device 300. The reference image data D2 is composed of image data of the reference images, which are images of each of the objects 810-870 without defects. The configuration of the reference image is the same as the configuration of the object area image defined by the object area information D1b (FIG. 5) for the corresponding object area. For example, the number of pixels in the first direction Dx, the number of pixels in the second direction Dy, and the color components of the color values ​​of each pixel are the same between the reference image and the object area image. The operator generates the reference image data D2 in advance using captured image data of the label sheet 800 without defects. Alternatively, the processor 210 of the first data processing device 200 may generate the reference image data D2 using the object region image data of each object region acquired in S135 of FIG.

[0051] The worker also places the multifunction device 900 to be inspected and the digital camera 110, as shown in Figure 2. The label sheet 800 of the multifunction device 900 may have a defect. The worker connects the digital camera 110 to the communication interface 370 of the second data processing device 300 (Figure 1). The worker then operates the operation unit 350 to input an instruction to start the inspection process. In response to the instruction, the processor 310 executes the inspection process in accordance with the inspection program 331.

[0052] Steps S310 and S320 are the same as steps S110 and S120, respectively, in Figure 4. The processor 310 uses image data from the digital camera 110 to acquire photographed image data of the label sheet 800.

[0053] In S330, processor 310 refers to object region information D1b of correspondence data D1 (FIG. 5). For each of the K object regions defined by correspondence data D1, processor 310 acquires object region image data representing an image of the object region from the captured image data.

[0054] In S350, the processor 310 selects an unprocessed object region from the K object regions as an object region of interest.

[0055] In S360, processor 310 calculates the difference Vd between the image of the target object region (referred to as the target object image) and the reference image corresponding to the target object region. The reference image is an image represented by reference image data D2 and is an image of an object without defects. The difference Vd may be various values ​​indicating the difference between the target object image and the reference image. In this embodiment, processor 310 calculates the difference Vd as follows: For each pixel, processor 310 calculates the three absolute values ​​of the three differences in the three RGB component values ​​between the target object image and the reference image. For each pixel, processor 310 calculates a total difference value, which is the sum of the three absolute values ​​of the three RGB differences. Processor 310 calculates the average of the total difference values ​​for all pixels in the target object region as the difference Vd. The difference Vd is larger the greater the difference between the target object image and the reference image. Note that instead of the above value, the difference Vd may be various values ​​indicating the difference between the target object image and the reference image, such as the L2 norm using the differences between each of the RGB components.

[0056] In S370, processor 310 refers to correspondence data D1 and acquires object type D1c of the target object region. Then, processor 310 sets reference value Vx indicating the conditions for inspection to a value previously associated with object type D1c (S371, S372, S373). The correspondence between object type D1c and reference value Vx is as follows: First image type: Vx=Vx1 Second image type: Vx=Vx2 Character type: Vx=Vx3

[0057] These values ​​Vx1, Vx2, and Vx3 are experimentally determined in advance so that the difference Vd is equal to or greater than the reference value Vx when the image of the object in the target object region contains unacceptable defects. In this embodiment, Vx2 > Vx3 > Vx1. The first reference value Vx1 for the first image type including an authentication mark allows a relatively small difference Vd, while the second reference value Vx2 for the second image type including a photograph allows a relatively large difference Vd. The third value Vx3 for the character type is a value between these two.

[0058] In S380, processor 310 determines whether difference Vd is less than reference value Vx. If difference Vd is equal to or greater than reference value Vx (S380: No), the test result is a failure. In this case, in S410, processor 310 displays information indicating a failure test result on display unit 340. Then, processor 310 ends the processing of FIG. 7.

[0059] If the difference Vd is less than the reference value Vx (S380: Yes), the inspection result is a pass. In this case, in S390, processor 310 determines whether all object regions have been processed. If unprocessed object regions remain (S390: No), processor 310 proceeds to S350 and processes the unprocessed object regions. If the inspection results for all object regions are pass (S390: Yes), in S400, processor 310 displays information indicating a pass inspection result on display unit 340. Then, processor 310 ends the processing of FIG. 7.

[0060] The processes of S400 and S410 may be any processes associated with the test results. For example, the processor 310 may store test result data indicating the test results in the storage device 315 (e.g., the non-volatile storage device 330). In this way, the processor 310 may execute an output process to output the test result data to a storage device or a display device.

[0061] As described above, in this embodiment, the processor 210 of the first data processing device 200 (FIG. 1) generates correspondence data D1 for inspecting the appearance of a label sheet 800, which is an example of an object, in the generation process of FIG. 4. In S130, the processor 210 uses the trained object detection model M1 to detect K object regions (in this embodiment, regions surrounded by bounding boxes BB1-BB7) corresponding to the K objects 810-870 from captured image data representing a captured image of the label sheet 800 containing K objects (in this embodiment, K=7). In S135-S180, the processor 210 generates correspondence data D1 (FIG. 5) indicating K correspondences corresponding to the K object regions, respectively. As shown in FIG. 5, each of the K correspondences indicates a correspondence between object region information D1b and an object type D1c (i.e., condition information D1c). Object area information D1b (coordinates D1b1, D1b2 in this embodiment) is an example of information that defines the object area in the captured image of label sheet 800. As explained in S370 of FIG. 7, condition information D1c indicates the inspection condition that corresponds to the type of object area among L (in this embodiment, L=3) inspection conditions (specifically, reference values ​​Vx1, Vx2, Vx3). In this way, object type D1c is an example of condition information that indicates the inspection condition. Then, in S190 of FIG. 4, processor 210 stores correspondence data D1 in storage device 215.

[0062] This configuration ensures that the correspondence data D1 used to determine the inspection conditions (here, the reference value Vx) for the object regions when inspecting the label sheet 800 (FIG. 7) is generated appropriately. If an operator were to determine information indicating the coordinates and type of each object region and input that information into the first data processing device 200, the burden on the operator would be significant. In particular, when the label sheet 800 contains multiple objects, as in the embodiment of FIG. 3(A), it is not easy for the operator to determine the correct correspondence between each of the multiple objects. In this embodiment, the processor 210 can generate appropriate correspondence data D1 without the operator having to determine the coordinates and type of the object regions. This significantly reduces the burden on the operator.

[0063] 6, in the process of generating correspondence data D1, processor 210 determines the type of object region (here, object type D1c) by analyzing the image of the object region based on predetermined rules. Processor 210 can generate appropriate correspondence data D1 based on the predetermined rules.

[0064] In this embodiment, as described in FIG. 7, the reference value Vx indicating the inspection condition is set to one of three values ​​Vx1, Vx2, and Vx3 depending on the type of object region. That is, the total number L of inspection conditions is 3, which is a value equal to or greater than 2. In S360 of FIG. 7, the processor 310 calculates the difference Vd. The difference Vd indicates the difference between the target object image, which is an image of the region indicated by the object region information D1b (FIG. 5) in the captured image for inspection, and the reference object image, which is an image of an object without abnormality that is pre-associated with the object region information D1b. As described in S380 of FIG. 7, the L inspection conditions (i.e., the L reference values ​​Vx1, Vx2, and Vx3) are conditions for determining that the difference Vd indicates that the appearance of the object represented by the target object image is normal. Here, a relatively large first reference value Vx1 allows a relatively large difference Vd, and a relatively small second reference value Vx2 allows a relatively small difference Vd. The third reference value Vx3 is a value between these two. Thus, the L inspection conditions include multiple inspection conditions that indicate different standards (in this embodiment, different reference values ​​Vx1, Vx2, and Vx3) for determining that the difference Vd indicates a normal appearance. In this embodiment, each object area is inspected according to the appropriate inspection condition associated with the type of object area, allowing for appropriate inspection of the label sheet 800. In the generation process of Figure 4, the processor 210 can appropriately generate the correspondence data D1 referenced in this inspection.

[0065] As shown in FIG. 6, the type of object region is one of L types, including a first image type that includes a mark established based on a standard or law. As described in S370 of FIG. 7, the first reference value Vx1 associated with the first image type is the smallest of the L reference values ​​Vx1, Vx2, and Vx3 associated with the L types. That is, the standard indicated by the inspection conditions associated with the first image type (here, the first reference value Vx1) is the standard that is most difficult to satisfy among the L standards indicated by the L inspection conditions associated with the L types (here, the reference values ​​Vx1, Vx2, and Vx3). Therefore, when a mark established based on a standard or law has a defect, the possibility of erroneously determining that the difference Vd of the mark does not indicate an appearance abnormality is reduced.

[0066] 6, the type of object region is one of L types, including a second image type that includes a photograph. As described in S370 of FIG. 7, the second reference value Vx2 associated with the second image type is the largest of the L reference values ​​Vx1, Vx2, and Vx3 associated with the L types. That is, the standard indicated by the inspection conditions associated with the second image type (here, the second reference value Vx2) is the standard most likely to be satisfied among the L standards indicated by the L inspection conditions associated with the L types (here, the reference values ​​Vx1, Vx2, and Vx3). Therefore, when a defect in a photograph is a small defect that should be tolerated, the likelihood that the difference Vd in the photograph will be determined to indicate an abnormality in appearance is reduced.

[0067] The type of object region is one of L types, including a character type including text and a second image type including a photograph. As described in S370 of FIG. 7, the third reference value Vx3 associated with the character type is smaller than the second reference value Vx2 associated with the second image type. That is, the standard indicated by the inspection conditions associated with the character type (here, the third reference value Vx3) is less likely to be met than the standard indicated by the inspection conditions associated with the second image type (here, the second reference value Vx2). This reduces the likelihood that the difference Vd between defective characters will be erroneously determined not to indicate an abnormality in appearance. Furthermore, when the defect in the photograph is a small defect that should be tolerated, the likelihood that the difference Vd between the photograph will be determined to indicate an abnormality in appearance is reduced.

[0068] B. Second Example: Figure 8 is a flowchart showing another embodiment of the object type determination process. At S160 of Figure 4, the processor 210 of the first data processing device 200 (Figure 1) executes the process of Figure 8 instead of the process of Figure 6. In this embodiment, the processor 210 determines the type of object region associated with the inspection condition (i.e., object type D1c (Figure 5)) using the type of object region determined by the object detection model M1 at S130 of Figure 4. In the example of Figure 8, the processor 210 determines the object type D1c to be one of the following five types: First image type: Type including marks established based on laws or standards Secondary image type: Type containing photographs and complex illustrations Third image type: Type containing simple illustrations First character type: A type that includes a character image consisting of a relatively small number of characters, such as a logotype Second character type: A type that includes text images consisting of a relatively large number of characters, such as explanatory text

[0069] The correspondence between the type determined by the object detection model M1 and the object type D1c is determined in advance. In this embodiment, it is as follows. Logotype 810: First character type Certification Mark 820: 1st Image Type Description 830: Second character type Trademark 840: Third Image Type First character string 850: First character type Photo 860: Second image type Second character string 870: First character type

[0070] In S210a, the processor 210 determines the object type D1c in accordance with the above correspondence relationship, and then the processor 210 ends the processing of FIG.

[0071] Fig. 9 is a flowchart showing another embodiment of the inspection process. The only difference from the embodiment of Fig. 7 is that S370 in Fig. 7 is replaced with S370a in Fig. 9. The other parts of the inspection process are the same as the corresponding parts in Fig. 7 (illustrations and descriptions of the same parts are omitted).

[0072] In S370a, the processor 310 of the second data processing device 300 (FIG. 1) refers to the correspondence data D1 and acquires the object type D1c of the target object region. Then, the processor 310 sets the reference value Vx indicating the condition for inspection to a value previously associated with the object type D1c (S371a-S375a). The correspondence between the object type D1c and the reference value Vx is as follows: First image type: Vx=Va1 Second image type: Vx=Va2 Third image type: Vx=Va3 First character type: Vx=Va4 Second character type: Vx=Va5

[0073] These values ​​Va1-Va5 are experimentally determined in advance so that the difference Vd is equal to or greater than the reference value Vx when the image of the object in the target object region contains unacceptable defects. In this embodiment, Va2>Va3=Va4>Va1=Va5. The first reference value Va1 for the first image type including the certification mark and the fifth reference value Va5 for the second character type including the explanatory text are the smallest of the five reference values ​​and allow a relatively small difference Vd. The second reference value Va2 for the second image type including a photograph is the largest of the five reference values ​​and allow a relatively large difference Vd. The third reference value Va3 for the third image type including a trademark and the fourth reference value Va4 for the first character type including a logotype are values ​​between them.

[0074] As described above, the generation process of this embodiment is the same as the generation process of the first embodiment (FIG. 4) except for the following two points. (1) The type determined by the object detection model M1 is used to determine the object type D1c. (2) The number of types of object type D1c is five. Therefore, this embodiment has the same various advantages as Embodiment 1. For example, the processor 210 can generate appropriate correspondence data D1 without the operator having to determine the coordinates and type of the object region.

[0075] 8, in the process of generating correspondence data D1, processor 210 determines the type of object region (here, object type D1c) by using trained object detection model M1 (FIG. 4: S130). Processor 210 can generate appropriate correspondence data D1 by using the detection result by object detection model M1.

[0076] In this embodiment, as described in FIG. 9, the reference value Vx indicating the inspection condition is set to one of five values ​​Va1-Va5 depending on the type of object region. That is, the total number of inspection conditions L is 5, which is a value equal to or greater than 2. The L inspection conditions include multiple inspection conditions that indicate different criteria for determining that the difference Vd indicates that the appearance is normal. In this embodiment, the criteria differ among the following three reference value groups: (Group 1) Va2 (Group 2) Va3, Va4 (3rd group) Va1, Va5 In this way, in this embodiment, the area of ​​each object is inspected according to the appropriate inspection conditions associated with the type of object area, enabling appropriate inspection of the label sheet 800. In the generation process of Figures 4 and 8, the processor 210 can appropriately generate the correspondence data D1 that is referenced in such an inspection.

[0077] As shown in FIG. 8 , the type of object region is one of L types, including a first image type that includes a mark established based on a standard or law. As described in S370a of FIG. 9 , the first reference value Va1 associated with the first image type is the smallest of the L reference values ​​Va1-Va5. That is, the first reference value Va1 associated with the first image type is the most difficult to satisfy of the L reference values ​​Va1-Va5 associated with the L types. Therefore, if a mark established based on a standard or law has a defect, the possibility of the difference Vd of the mark being erroneously determined not to indicate an abnormality in appearance is reduced. In this embodiment, the fifth reference value Va5 of the second character type that includes a description is the same as the first reference value Va1. Therefore, if the description has a defect, the possibility of the difference Vd of the description being erroneously determined not to indicate an abnormality in appearance is reduced. In this way, the multiple reference values ​​Va1, Va5 corresponding to the multiple types may be the smallest reference values ​​(that is, the reference values ​​that are most difficult to meet).

[0078] 8, the type of object region is one of L types, including a second image type that includes a photograph. As described in S370a of FIG. 9, the second reference value Va2 associated with the second image type is the largest of the L reference values ​​Va1-Va5. That is, the second reference value Va2 associated with the second image type is the reference value that is most likely to be satisfied of the L reference values ​​Va1-Va5. Therefore, when a defect in a photograph is a small defect that should be tolerated, the possibility that the difference Vd of the photograph will be determined to indicate an abnormality in appearance is reduced.

[0079] As shown in FIG. 8 , the object region type is one of L types, including a first character type and a second character type that include text, and a second image type and a third image type that include one or both of an illustration and a photograph. As described in S370a of FIG. 9 , the reference values ​​Va4 and Va5 associated with the first and second character types are smaller than the reference values ​​Va2 and Va3 associated with the second and third image types. That is, the reference values ​​Va4 and Va5 associated with the first and second character types are less likely to be satisfied than the reference values ​​Va2 and Va3 associated with the second and third image types. This reduces the likelihood that the difference Vd between characters with defects will be erroneously determined not to indicate an abnormality in appearance. This reduces the likelihood that the difference Vd between an illustration with a small, acceptable defect or the difference Vd between a photograph with a small, acceptable defect will be determined to indicate an abnormality in appearance.

[0080] C. Third Example: Fig. 10 is a flowchart showing another embodiment of the object type determination process. The processor 210 of the first data processing device 200 (Fig. 1) executes the process of Fig. 10 in place of the process of Fig. 6 at S160 of Fig. 4. In this embodiment, the processor 210 inputs image data of the target object region into a trained classification model, thereby determining the type of object region (i.e., object type D1c (Fig. 5)) associated with the inspection conditions. In this embodiment, the object type D1c is determined to be one of five types, as in the embodiment of Fig. 8. The correspondence between the objects 810-870 and the object type D1c is assumed to be the same as the correspondence in the embodiment of Fig. 8.

[0081] 11 is a schematic diagram showing an example of a classification model. In this embodiment, the classification model M3 is a program module that forms a convolutional neural network. Although not shown, data of the trained classification model M3 is pre-stored in the non-volatile storage device 230 of the first data processing device 200.

[0082] The classification model M3 has p (p is an integer equal to or greater than 1) convolutional layers V31-V3p and q (q is an integer equal to or greater than 1) fully connected layers N31-N3q (p is, for example, 2, and q is, for example, 3) following the convolutional layers V31-V3p. A pooling layer is provided immediately after one or more of the p convolutional layers V31-V3p. The classification model M3 generates output data M3o based on input image data M3i input to the classification model M3. The output data M3o indicates the degree of certainty of each of multiple types of object regions. The type associated with the greatest degree of certainty indicates the type of object region represented by the input image data M3i.

[0083] The convolutional layers V31-V3p perform processing on input data, including convolution processing using filters and bias addition processing. Each of the convolutional layers V31-V3p has a set of calculation parameters including multiple weights and multiple biases for multiple filters. The pooling layer performs processing to reduce the number of dimensions of data input from the previous convolutional layer. In this embodiment, the pooling layer performs max pooling. The fully connected layers N31-N3q reduce the dimension of data input from the previous layer. Each of the fully connected layers N31-N3q has a set of calculation parameters including multiple weights and multiple biases.

[0084] The data generated by each of the convolutional layers V31-V3p and the fully connected layers N31-N3q is input to an activation function and transformed. In this example, the last layer (here, the fully connected layer N3q) uses Softmax, and the other layers use ReLU.

[0085] The classification model M3 is trained using image data of multiple images representing any of the objects 810-870. A training processor (e.g., processor 210) uses the image data to perform calculations on each of layers V31-V3p and N31-N3q of the classification model M3, thereby generating output data M3o. The training processor then adjusts multiple calculation parameters of the classification model M3 so that the output data M3o approaches the correct data. For example, image data of an image of a logotype 810 is input to the classification model M3. A first character type is associated with the logotype 810. Therefore, multiple calculation parameters of the classification model M3 are adjusted so that the confidence level of the first character type indicated by the output data M3o is maximized. Similarly, for the other objects 820-870, multiple calculation parameters of the classification model M3 are adjusted so that the confidence level of the object type D1c associated with the object is maximized.

[0086] Various methods may be used to adjust the calculation parameters. In this embodiment, multiple calculation parameters of the classification model M3 are adjusted so that the loss calculated using a loss function is reduced. The loss function may be various functions (e.g., cross entropy, sum of squares error, etc.) that calculate an evaluation value of the difference between the output data M3o and the ground truth data associated with the input image data M3i input to the classification model M3. Furthermore, an algorithm that uses, for example, backpropagation and gradient descent may be adopted as an algorithm for adjusting the multiple calculation parameters. Here, so-called Adam optimization may be performed. The ground truth data is generated in advance by an operator.

[0087] In S210b (FIG. 10), processor 210 generates output data M3o by performing operations on each layer V31-V3p, N31-N3q of trained classification model M3 (FIG. 11) using object region image data of the target object region. The output data M3o indicates an appropriate object type D1c for the target object region. Processor 210 determines the object type D1c to be the type indicated by output data M3o. Then, processor 210 ends the processing of FIG. 10.

[0088] As described above, in this embodiment, in the process of generating correspondence data D1, processor 210 determines the type of object region (here, object type D1c) by using classification model M3 trained to classify the type of object region. Processor 210 can generate appropriate correspondence data D1 using the classification result by classification model M3.

[0089] The generation process of this embodiment is the same as that of the second embodiment (FIG. 4), except that classification model M3 is used to determine object type D1c. Therefore, this embodiment has the same various advantages as the second embodiment.

[0090] D. Fourth Example: 12 is a flowchart showing another embodiment of the process for generating correspondence data D1. In this embodiment, S510-S560 are inserted between S180 and S190 in FIG. 4. If the determination result of S180 is Yes, the process proceeds to S510. If the determination result of S180 is No, the process proceeds to S150. This embodiment is applicable to each of the first to third embodiments described above.

[0091] Marks established based on laws or standards may be displayed together with other objects. For example, a "CE mark" may be displayed together with a "character string representing the country of manufacture." In this case, it is preferable that inspection conditions equivalent to those for the "CE mark" be applied to the "character string representing the country of manufacture." In this embodiment, the certification mark 820 ( FIG. 3(A) ) is the "CE mark." Specific condition information (also referred to as a specific type) associated with specific inspection conditions is added to multiple types selectable as the object type D1c ( FIG. 5 ). The specific inspection conditions indicate the same criteria as those for the "CE mark" and are applied to the "character string representing the country of manufacture" displayed together with the "CE mark." In this embodiment, the second character string 870 is the "character string representing the country of manufacture." Note that if the label sticker does not include the "CE mark," the object type D1c for the "character string representing the country of manufacture" is the same as the object type D1c in the above embodiments (e.g., the character type in FIG. 6 or the first character type in FIG. 8 ).

[0092] In S510 (FIG. 12), processor 210 searches for an object region representing a "CE mark" from among the multiple object regions detected in S130 (FIG. 4). Any method may be used for the search. For example, pattern matching using image data of a "CE mark" prepared in advance may be employed. Hereinafter, the object searched for in S510 is also referred to as a first object. The "CE mark" is an example of a first object.

[0093] In S520, processor 210 determines whether or not a first object region representing the first object is found. If a first object region is not found (S520: No), processor 210 proceeds to S190 (FIG. 4).

[0094] If the first object area is found (S520: Yes), in S530, processor 210 searches the multiple object areas for an object area that represents a "character string representing the country of manufacture (in this embodiment, second character string 870)." Any method for the search may be used. For example, pattern matching using image data of the "MADE IN" character string prepared in advance may be employed. Hereinafter, the object searched for in S530 will also be referred to as a second object. The "character string representing the country of manufacture" is an example of a second object.

[0095] In S540, processor 210 determines whether a second object region representing the second object is found. If a second object region is not found (S540: No), processor 210 proceeds to S190 (FIG. 4).

[0096] If a second object region is found (S540: Yes), then in S550, processor 210 determines that the type of the second object region is a specific type. Then, in S560, processor 210 changes the data indicating the object type D1c of the second object region in correspondence data D1 (FIG. 5) to data indicating a specific type. Then, processor 210 proceeds to S190 (FIG. 4). Thus, if the label sheet contains a "CE mark," the object type D1c of the "character string indicating country of manufacture" is set to a specific type. While not shown, during the inspection process, processor 310 of second data processing device 300 sets the inspection condition criteria for the "character string indicating country of manufacture" to the criteria associated with the specific type (in this embodiment, the same as the inspection condition criteria for the "CE mark").

[0097] As described above, in the generation process of this embodiment, processor 210 performs the following steps. At S510 (FIG. 12), processor 210 searches for a first object region representing a predetermined first object (in this embodiment, the authentication mark 820) from among the K object regions included on the label sheet. If a first object region is found (S520: Yes), processor 210 searches for a second object region representing a predetermined second object (in this embodiment, the second character string 870) from among the K object regions (S530). If a second object region is found (S540: Yes), processor 210 sets condition information D1c, which is associated with object region information D1b (FIG. 5) that defines the second object region, to specific condition information associated with predetermined specific inspection conditions (S550-S560). With this configuration, when a label sheet contains a first object and a second object, processor 210 can generate correspondence data D1 that associates specific inspection conditions with second object regions. In this way, processor 210 can flexibly determine the correspondence between object regions and condition information D1c depending on the number of objects included on the label sheet.

[0098] E. Fifth Example: FIG. 13 is a schematic diagram showing another example of a label sheet. Unlike the label sheet 800 of FIG. 3A, the objects 820, 830, 850, and 870 of FIG. 3A are replaced with objects 820e, 830e, 850e, and 870e in the label sheet 800e. The authentication mark 820e is a different mark from the authentication mark 820 of FIG. 3A. The explanatory text 830e provides information different from that of the explanatory text 830 of FIG. 3A. The first character string 850e represents a model number different from that of the first character string 850 of FIG. 3A. The second character string 870e represents a country of manufacture different from that of the second character string 870 of FIG. 3A. The photograph 860 is omitted. The position of the explanatory text 830e in the label sheet 800e is different from that of the explanatory text 830 in the label sheet 800 (FIG. 3A). Hereinafter, the label sheet 800 in Figure 3(A) will be referred to as the first-type label sheet 800, and the label sheet 800e in Figure 13 will be referred to as the second-type label sheet 800e. Additionally, the correspondence data D1 for the first-type label sheet 800 will be referred to as the first correspondence data D1.

[0099] Multiple different label sheets 800, 800e can be inspected. In this case, the first data processing device 200 (FIG. 1) generates second correspondence data for the second-type label sheet 800e in addition to the first correspondence data D1 for the first-type label sheet 800. The second correspondence data, like the first correspondence data D1 (FIG. 5), indicates the correspondence between object numbers D1a, object area information D1b, and condition information D1c.

[0100] FIG. 14 is a flowchart showing an example of the generation process. In S610, the processor 210 generates first correspondence data D1 for the first-type label sheet 800. The generation process of S610 may be the same as the generation process of an embodiment arbitrarily selected from the multiple embodiments described above. In S620, the processor 210 generates second correspondence data for the second-type label sheet 800e. The algorithm for the generation process of S620 may be the same as the algorithm for the generation process of S610. The object detection model M1 (FIG. 1) is pre-trained to detect objects 810, 820e, 830e, 840, 850e, and 870 included in the second-type label sheet 800e in addition to objects 810-870 included in the first-type label sheet 800. In S620, the processor 210 generates the second correspondence data using the captured image data of the second-type label sheet 800e.

[0101] As described above, in addition to generating the first correspondence data D1, the processor 210 also generates second correspondence data for the second-type label sheet 800e. The process for generating second correspondence data includes the process shown in FIG. 4. In S130, the processor 210 uses the trained object detection model M1 to detect T object regions corresponding to the T objects 810, 820e, 830e, 840, 850e, and 870 from captured image data representing a captured image of the second-type label sheet 800e containing T objects (T=6 in this embodiment) and without any abnormalities in appearance. In S135-S180, the processor 210 generates second correspondence data indicating T correspondences corresponding to the T object regions. Each of the T correspondences indicates a correspondence between object region information D1b and object type D1c, similar to the correspondences shown in FIG. 5. Object area information D1b (specifically, coordinates D1b1, D1b2) is an example of information that defines an object area in a captured image of the second-type label sheet 800e. Similarly to object type D1c in the first correspondence data D1, object type D1c in the second correspondence data indicates an inspection condition that corresponds to the type of object area among U inspection conditions (U is an integer between 1 and T, for example, U=L). Thus, object type D1c is an example of condition information that indicates an inspection condition. Then, in S190 of FIG. 4, the processor 210 stores the second correspondence data in the storage device 215.

[0102] This configuration appropriately generates second correspondence data that is referenced to determine the inspection conditions (here, the reference value Vx) for object areas when inspecting the second-type label sheet 800e. If an operator were to determine information indicating the coordinates and type of each object area and input that information into the first data processing device 200, the burden on the operator would be significant. In particular, when multiple label sheets 800, 800e each contain multiple objects, as in the embodiments of Figures 3(A) and 13, it is not easy for the operator to determine the correct correspondence between each of the multiple objects. In this embodiment, the processor 210 can generate appropriate correspondence data for multiple label sheets 800, 800e without the operator having to determine the coordinates and type of the object areas. This significantly reduces the burden on the operator.

[0103] F. Variations: (1) The process of determining the type of object region based on a predetermined rule may be various other processes instead of the process of FIG. 6. For example, processor 210 may calculate the variance of the component values ​​for each color component. Then, processor 210 may determine whether a second condition is satisfied, that is, all variances of all color components are equal to or greater than a variance threshold. If the second condition is satisfied, processor 210 may proceed to S260, and if the second condition is not satisfied, processor 210 may proceed to S250. Alternatively, processor 210 may analyze the image in more detail to determine the type of object region from a greater number of options, as in the embodiment of FIG. 8.

[0104] (2) The classification model for classifying the types of object regions may have various other configurations that can show the correspondence between image data and types, instead of the configuration shown in Figure 11. For example, the classification model may be configured with multiple fully connected layers.

[0105] (3) In the example of FIG. 12, the first object area searched in S510 and the second object area searched in S530 may be areas representing any other object instead of the "CE mark" and the "character string representing the country of manufacture." For example, the first object area may be the "GS mark," and the second object area may be an "explanatory text explaining precautions required by laws and regulations." In either case, the inspection conditions associated with the second object area may be any conditions. For example, the defects permitted by the inspection conditions associated with the second object area may be smaller than the defects permitted by the inspection conditions associated with the first object area.

[0106] (4) The object region information D1b in the correspondence data D1 (FIG. 5) may be any information that defines the object region in the captured image, instead of the coordinates D1b1 and D1b2. For example, the object region information D1b may indicate the coordinates of the center of the object region, the length of the object region in the first direction Dx, and the length of the object region in the second direction Dy. Various information that defines the parts of the captured image that are included in the object region and the parts that are not included in the object region can be used as the object region information D1b.

[0107] Furthermore, the condition information D1c may be any information indicating the inspection conditions instead of the identification number of the type of object region. For example, the condition information D1c may indicate the reference value Vx. It can be said that the identification number of the type of object region indicates the identification number of the inspection conditions.

[0108] (5) The inspection process may be various processes for inspecting an object containing multiple objects (e.g., label sheets 800, 800e) instead of the processes shown in Figures 7 and 9. For example, the calculation formula for the difference Vd may be various formulas for calculating an evaluation value of the difference between the target object image and the reference image corresponding to the target object region. For example, the difference Vd may be the total number of different-color pixels, which are pixels that exhibit a difference in color value equal to or greater than a predetermined reference value when comparing the color values ​​of each pixel between two images. The difference in color value may be various values ​​that increase as the visual color difference increases. For example, the difference in color value may be the sum of the three absolute values ​​of the three RGB differences. The difference Vd may also be the ratio of different-color pixels to all pixels in the image. The difference Vd may be various values ​​that indicate the magnitude of the visual difference between the target object image and the reference image.

[0109] (6) The total number L of inspection conditions applied to the first-type label sheet 800 may be any integer greater than or equal to 1 and less than or equal to K (K is the total number of objects included in the first-type label sheet 800). When L is 2 or greater, multiple inspection conditions are used for inspection, allowing for proper inspection of the first-type label sheet 800. The total number K of objects may be any integer greater than or equal to 1. When K is 2 or greater, the first-type label sheet 800 can represent various pieces of information using multiple objects. Similarly, the total number U of inspection conditions applied to the second-type label sheet 800e may be any integer greater than or equal to 1 and less than or equal to T (T is the total number of objects included in the second-type label sheet 800e). When U is 2 or greater, multiple inspection conditions are used for inspection, allowing for proper inspection. The total number T of objects may be any integer greater than or equal to 1. When T is 2 or greater, the second-type label sheet 800e can represent various pieces of information using multiple objects. Here, U may be different from L. T may be different from K. In either case, the numbers K, L, T, and U are predetermined.

[0110] The L reference values ​​corresponding to the L inspection conditions may be various values. For example, in the embodiment of FIG. 9, the two reference values ​​Va1 and Va5 corresponding to the two inspection conditions are the smallest reference values ​​(i.e., the reference values ​​least likely to be met). The multiple reference values ​​corresponding to the multiple inspection conditions (e.g., Va2 and Va4) may be the largest reference values ​​(i.e., the reference values ​​most likely to be met). In this way, the multiple reference values ​​for the multiple inspection conditions may be the same. Alternatively, the L reference values ​​for the L inspection conditions may be different from one another. The U reference values ​​corresponding to the U inspection conditions may also be various values.

[0111] (7) The types of object areas associated with the inspection conditions may be of various types. The types of object areas may include one or more of the following: a first type including a mark established based on a standard or law; a second type including a photograph; and a third type including text. The types of object areas may also include a fourth type including a photograph or both.

[0112] (8) When comparing the likelihood of satisfying multiple inspection conditions, it is preferable to use the difference Vd normalized according to the image size in order to mitigate the effect of differences in image size (i.e., the number of pixels). For example, in the above embodiment, the difference Vd is the average value of the total difference values ​​of all pixels, and the effect of size on the difference Vd is mitigated. Therefore, if the difference Vd is the same and the inspection result based on the first inspection condition is pass, and the inspection result based on the second inspection condition is fail, the second inspection condition is more difficult to satisfy than the first inspection condition.

[0113] (9) The object detection model may be any other model instead of the YOLO model. For example, the object detection model may be an improved YOLO model such as "YOLO v3." Other models, such as SSD R-CNN, Fast R-CNN, Faster R-CNN, and Mask R-CNN, may also be used.

[0114] (10) The object of inspection is not limited to a label sheet affixed to a product (e.g., a multifunction printer, a sewing machine, a cutting machine, a mobile terminal, etc.), but may be any object containing one or more objects. For example, the object may be a label image printed directly on the product. The object may also be any part of the product, such as a tag or accessory attached to the product. In either case, the object included in the object is not limited to objects represented by two-dimensional images such as marks and character strings, but may also include three-dimensional objects having a three-dimensional shape. Such objects can also be inspected using captured image data.

[0115] (11) The color components of the image data used to generate the correspondence data (e.g., the correspondence data D1 in FIG. 5) and to inspect the object are not limited to the three color components of RGB, but may be any color components. For example, monochrome image data indicating only luminance values ​​may be used. Furthermore, image data having four color components of CMYK may be used.

[0116] (12) A generating device that executes a generating process (e.g., the generating process of FIG. 4) may be a device of a different type (e.g., a digital camera, a scanner, a smartphone, a server) from a personal computer (e.g., data processing device 200 (FIG. 1)). Furthermore, multiple devices (e.g., computers) that can communicate with each other via a network may share some of the functions of the generating process and collectively provide the functions of the generating process (a system including these devices corresponds to a generating device). The same applies to an inspection device that executes an inspection process (e.g., the inspection process of FIG. 7). Furthermore, the same device may execute both the generating process and the inspection process.

[0117] In each of the above embodiments, a part of the configuration realized by hardware may be replaced by software, and conversely, a part or all of the configuration realized by software may be replaced by hardware. For example, the determination process of FIG. 6 may be realized by a dedicated hardware circuit.

[0118] Furthermore, when some or all of the functions of the present invention are realized by a computer program, the program can be provided in a form stored on a computer-readable recording medium (e.g., a non-transitory recording medium). The program can be used while stored on the same or a different recording medium (computer-readable recording medium) from when it was provided. The "computer-readable recording medium" is not limited to portable recording media such as memory cards and CD-ROMs, but can also include internal storage devices within a computer, such as various ROMs, and external storage devices connected to a computer, such as a hard disk drive.

[0119] The present invention has been described above based on examples and modifications, but the above-described embodiments of the invention are intended to facilitate understanding of the present invention and are not intended to limit the present invention. The present invention may be modified or improved without departing from the spirit of the invention, and equivalents thereof are also included within the scope of the present invention. [Explanation of symbols]

[0120] 110...digital camera, 190...support base, 191...top surface, 200...first data processing device, 210...processor, 215...storage device, 220...volatile storage device, 230...non-volatile storage device, 231...generation program, 240...display unit, 250...operation unit, 270...communication interface, 300...second data processing device, 310...processor, 315...storage device, 320...volatile storage device, 330...non-volatile storage device, 331...inspection program, 340...display unit, 350...operation unit, 370...communication interface, 900...multifunction device, 901...first side surface, 909...bottom surface, 1000...data processing system

Claims

1. 1. A computer program for generating data for inspecting the appearance of an object, comprising: a first region detection function that uses a trained object detection model to detect K object regions corresponding to the K objects from first captured image data representing a first captured image of a first type of object that includes K objects (K is an integer equal to or greater than 1) and has no abnormalities in appearance; a first data generation function that generates first correspondence data indicating K correspondences corresponding to the K object regions, each of the K correspondences indicating a correspondence between object region information, which is information that defines an object region in a captured image of the first type object, and condition information indicating an inspection condition, and the condition information indicating an inspection condition that is associated with the type of the object region out of L inspection conditions (L is an integer of 1 or more and K or less); a first data storage function for storing the first correspondence data in a storage device; A computer program that enables a computer to realize the above.

2. 2. The computer program of claim 1, the first data generation function is configured to determine the type of the object region by analyzing an image of the object region based on a predetermined rule. Computer program.

3. 2. The computer program of claim 1, the first data generation function is configured to determine the type of the object region by using the trained object detection model or a classification model trained to classify types of object regions. Computer program.

4. A computer program according to any one of claims 1 to 3, L is 2 or more, The L inspection conditions are conditions for determining that a difference between a target object image, which is an image of a region indicated by the object region information in a captured image for inspection, and a reference object image, which is an image of an object without anomalies that is pre-associated with the object region information, indicates that the appearance of the object represented by the target object image is normal; the L inspection conditions include a plurality of inspection conditions that indicate mutually different criteria for determining that the difference indicates that the appearance is normal; Computer program.

5. 5. A computer program according to claim 4, the type of the object region is one of L types including a first type including a mark established based on a standard or law, the criterion indicated by the inspection condition associated with the first type is the criterion that is most difficult to satisfy among the L criteria indicated by the L inspection conditions associated with the L types; Computer program.

6. 6. A computer program according to claim 4 or 5, the type of the object region is one of L types including a second type including a photograph, the criterion indicated by the inspection condition associated with the second type is the criterion that is most likely to be satisfied among the L criteria indicated by the L inspection conditions associated with the L types; Computer program.

7. 6. A computer program according to claim 4 or 5, the type of the object region is one of L types including a third type including text and a fourth type including one or both of an illustration and a photograph, The criteria indicated by the inspection conditions associated with the third type are less likely to be satisfied than the criteria indicated by the inspection conditions associated with the fourth type. Computer program.

8. 8. A computer program according to any one of claims 1 to 7, further comprising: a first search function for searching the K object regions for a first object region representing a predetermined first object; a second search function for searching the K object regions for a second object region representing a predetermined second object when the first object region is found; This is realized on a computer, the first data generation function is configured to set, when the second object area is found, condition information associated with object area information defining the second object area to specific condition information associated with predetermined specific inspection conditions. Computer program.

9. 9. A computer program according to any one of claims 1 to 8, further comprising: a second region detection function that uses the trained object detection model to detect T object regions corresponding to the T objects from second captured image data representing a second captured image of a second type of object that includes T objects (T is an integer equal to or greater than 1) and has no abnormal appearance; a second data generation function that generates second correspondence data indicating T correspondences corresponding to the T object regions, each of the T correspondences indicating a correspondence between object region information, which is information that defines an object region in a captured image of the second type object, and condition information indicating an inspection condition, and the condition information indicating an inspection condition that is associated with the type of the object region out of U inspection conditions (U is an integer of 1 or more and T or less); a second data storage function for storing the second correspondence data in a storage device; A computer program that enables a computer to realize the above.

10. A generating device for generating data for inspecting the appearance of an object, comprising: a first region detection unit that uses a trained object detection model to detect K object regions corresponding to the K objects from first captured image data representing a first captured image of a first type of object that includes K objects (K is an integer equal to or greater than 1) and has no abnormalities in appearance; a first data generation unit that generates first correspondence data indicating K correspondences corresponding to the K object regions, each of the K correspondences indicating a correspondence between object region information, which is information that defines an object region in a captured image of the first type object, and condition information that indicates an inspection condition, the condition information indicating an inspection condition that is associated with the type of the object region out of L inspection conditions (L is an integer that is equal to or greater than 1 and equal to or less than K); a first data storage unit that stores the first correspondence data in a storage device; A generating device comprising:

11. 1. A method for generating data for inspecting the appearance of an object, comprising: a first region detection step of detecting K object regions corresponding to the K objects (K is an integer equal to or greater than 1) from first captured image data representing a first captured image of a first type of object having no abnormalities in appearance, using a trained object detection model; a first data generation step of generating first correspondence data indicating K correspondences corresponding to the K object regions, each of the K correspondences indicating a correspondence between object region information, which is information defining an object region in a captured image of the first type object, and condition information indicating an inspection condition, the condition information indicating an inspection condition that is associated with the type of the object region out of L inspection conditions (L is an integer of 1 or more and K or less); a first data storage step of storing the first correspondence data in a storage device; A generating method comprising:

Images