Computer program, task generation device, and task generation method
The computer program and task generation device address the challenge of adapting image processing to varying inspection standards by using dedicated learning models, enabling efficient and accurate inspection across different sites without specialized engineering.
Patent Information
- Application Number
- JP2023075776
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-01
- Publication Date
- 2024-11-14
AI Technical Summary
Existing image processing systems require specialized engineers with programming and image processing skills, struggle with complex textures and gradation images, and face challenges in constructing algorithms that adapt to varying inspection standards across different sites.
A computer program and task generation device that utilizes a series of learning models to generate multiple tasks for inspecting targets, including object detection, classification, and defect detection, without the need for specialized engineering, and adapts to varying inspection standards by using dedicated learning models optimized for each site.
Enables inspection of targets with varying standards across different sites without requiring specialized engineers, simplifying the process and improving accuracy through dedicated learning models tailored to specific inspection needs.
Smart Images

Figure 2024160599000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a computer program, a task generation device, and a task generation method. [Background technology]
[0002] Patent Document 1 discloses an image inspection device that determines the pass / fail quality of an inspection object based on an image captured of the inspection object, and that implements a rule-based inspection mode that allows adjustment of parameters of an image processing tool, and a learning-based inspection mode that uses a classifier generated based on images of good and defective products. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2020-187070 A Summary of the Invention [Problem to be solved by the invention]
[0004] In general, rule-based image processing requires specialized engineers with programming and image processing skills, and the detection limit tends to depend on the skills of the engineers. In addition, it is difficult to develop image processing algorithms for images with complex textures or gradations, and the cost of changing image processing algorithms depending on the type of inspection object is high. Furthermore, image processing algorithms cannot be developed unless the inspection specifications are clear.
[0005] In addition, when inspecting an object using a learning model, an engineer with image processing skills is not required, but since the detection basis for the detection results of the learning model is a black box, it is difficult to quantify the inspection criteria for the inspection part. For this reason, it is often not usable for actual inspections where the inspection criteria differ depending on the inspection part of the object.
[0006] The present invention has been made in consideration of the above circumstances, and aims to provide a computer program, a task generation device, and a task generation method that enable inspection of an object for which inspection standards differ depending on the inspection part. [Means for solving the problem]
[0007] The present application includes multiple means for solving the above-mentioned problems. As one example, a computer program causes a computer to execute a process of acquiring an object image relating to an inspection object, and when the image of the inspection object is input, inputting the acquired object image into a task generation model which generates multiple tasks for inspecting the inspection object, and generating multiple tasks for inspecting the inspection object in the target image. Effect of the Invention
[0008] According to the present invention, it is possible to perform an examination on an examination subject in which the examination criteria differ depending on the examination site. [Brief description of the drawings]
[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a configuration of a task generation device according to the present embodiment. [Diagram 2] FIG. 2 is a diagram illustrating a first example of a processing scheme of a task generation device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a configuration of a first language generation model. [Figure 4] FIG. 4 is a diagram illustrating an example of an input image. [Diagram 5] FIG. 5 is a diagram showing an example of a caption for the input image shown in FIG. 4. [Figure 6] FIG. 2 is a diagram illustrating an example of a configuration of a second language generation model. [Figure 7] FIG. 13 is a diagram showing an example of hearing contents (text). [Figure 8] FIG. 8 is a diagram showing an example of a summary of the hearing contents shown in FIG. 7. [Figure 9]5 is a diagram showing an example of a summary of the hearing contents regarding the inspection target shown in the input image of FIG. 4. [Figure 10] FIG. 6 is a diagram showing an example of a corrected caption obtained by correcting the caption in FIG. 5. [Figure 11] FIG. 2 is a diagram illustrating an example of a configuration of an image generation model. [Figure 12] FIG. 2 is a diagram showing an example of a target image. [Figure 13] FIG. 2 is a diagram illustrating an example of a configuration of a sentence generation model. [Figure 14] FIG. 13 is a diagram showing an example of a sentence (before evaluation) regarding an inspection target. [Figure 15] FIG. 13 is a diagram showing an example of a sentence (after evaluation) regarding the inspection target. [Figure 16] FIG. 13 is a diagram showing an example of a process of converting a sentence related to an inspection target into a directed graph using a sentence task generation model. [Figure 17] FIG. 2 is a diagram showing an example of a plurality of tasks generated by a text task generation model (task generation model). [Figure 18] FIG. 13 is a diagram illustrating an example of the functions of a learning model dedicated to each task. [Figure 19] FIG. 11 is a diagram illustrating an example of processing of a defect image generation model. [Figure 20] FIG. 13 is a diagram illustrating a second example of a processing scheme of the task generation device. [Figure 21] FIG. 13 is a diagram illustrating a third example of a processing scheme of the task generation device. [Figure 22] FIG. 13 is a diagram illustrating an example of a task creation screen. [Diagram 23] FIG. 13 is a diagram showing an example of an examination setting screen. [Figure 24] FIG. 13 is a diagram illustrating an example of a task annotation screen. [Diagram 25] FIG. 13 is a diagram showing an example of a normal product image annotation screen. [Figure 26] FIG. 13 is a diagram showing an example of a target image annotation screen (before annotation). [Figure 27] FIG. 13 is a diagram showing an example of a target image annotation screen (after annotation). [Figure 28] FIG. 13 illustrates an example of a task flow selection screen. [Figure 29] FIG. 2 illustrates an example of a processing procedure of a task generation device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present invention will be described. Fig. 1 is a diagram showing an example of the configuration of a task generation device 100 of this embodiment. The task generation device 100 includes a control unit 10 that controls the entire device, a communication unit 11, a memory 12, a display unit 13, an operation unit 14, a storage unit 15, a processing unit 20, and a learning model unit 30. The processing unit 20 includes a divided region detection function 21, an integration function 22, and an evaluation function 23. The learning model unit 30 includes a first language generation model 40, a second language generation model 50, an image generation model 60, a sentence generation model 70, a task generation model 80, and a defect image generation model 90. The task generation model 80 includes a sentence task generation model 81.
[0011] The control unit 10 is configured by incorporating a required number of central processing units (CPUs), micro-processing units (MPUs), graphics processing units (GPUs), general-purpose computing on graphics processing units (GPGPUs), etc. The control unit 10 may also be configured by combining digital signal processors (DSPs), field-programmable gate arrays (FPGAs), etc.
[0012] The communication unit 11 includes a communication module and has a communication function with an external device (not shown). The communication unit 11 can receive an image (input image) of a product or part to be inspected, and an image (normal product image) of a normal product that is the same product or part as the inspection target and has no defects, from an external device or data server, and store the received image in the storage unit 15. The communication unit 11 can also receive a file or the like that records the contents of an interview with the inspection target from an external device or data server, and store the received file or the like in the storage unit 15. The details of the interview will be described later.
[0013] The memory 12 can be configured with a semiconductor memory such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory.
[0014] The display unit 13 can be configured with a liquid crystal display, an organic EL display, etc. Instead of the display unit 13, an external display device may be connected to the task generation device 100.
[0015] The operation unit 14 is configured with, for example, a keyboard, a mouse, a touch pad, or a touch panel, and can accept operations on the information displayed on the display unit 13.
[0016] The storage unit 15 may be configured with, for example, a hard disk or a semiconductor memory, and may store a computer program (program product) 16 and necessary information. The storage unit 15 may store a learning model unit 30.
[0017] The computer program 16 is deployed in the memory 12 under the control of the control unit 10 and executed by the control unit 10. The computer program 16 may be downloaded from an external device via the communication unit 11 and stored in the storage unit 15. Also, the computer program 16 recorded on a recording medium (e.g., an optically readable disk storage medium such as a CD-ROM) may be read by a recording medium reading unit and stored in the storage unit 15. The computer program 16 may be deployed to be executed on a single computer, or on multiple computers located at one site, or distributed across multiple sites and interconnected by a communication network.
[0018] The processing unit 20 and the learning model unit 30 will be described in detail later.
[0019] The task generation device 100 (control unit 10) acquires the input image of the inspection object and the hearing contents related to the inspection object, and generates a plurality of tasks for inspecting the inspection object by performing processing by the processing unit 20 and the learning model unit 30. The hearing contents related to the inspection object are not essential, but it is preferable to acquire the hearing contents in order to complement knowledge outside the image, such as what kind of inspection thinking is used to inspect the inspection object, which is difficult to infer from the input image alone. In this specification, "multiple tasks" refers to a series of tasks that indicate the flow of an inspection procedure according to an inspection thinking that shows how to perform an inspection depending on the inspection object. A "task" is an individual process in an inspection procedure. Specifically, the "task" corresponds to a process of detecting an inspection object from an image, a process of classifying inspection object parts from the detected inspection object, a process of detecting defects (defect candidates) in the classified inspection object parts, a process of determining whether or not the detected defects (defect candidates) are defective, and the like.
[0020] Next, a processing scheme for generating multiple tasks will be described.
[0021] 2 is a diagram showing a first example of a processing scheme of the task generation device 100. When the control unit 10 inputs an input image of an inspection target to the first language generation model 40, the first language generation model 40 generates a caption for the input image and outputs the generated caption to the integration function 22. When the control unit 10 inputs hearing contents related to the inspection target to the second language generation model 50, the second language generation model 50 generates a summary of the hearing contents and outputs the generated summary to the integration function 22 and the evaluation function 23. The integration function 22 corrects the input caption based on the summary and outputs the corrected caption to the image generation model 60.
[0022] When the control unit 10 inputs a normal product image (an image of a normal product without defects that is the same product or part as the inspection target) to the division area detection function 21, the division area detection function 21 generates division area information and outputs the generated division area information to the image generation model 60. The division area information includes information used as an auxiliary when segmenting the inspection target. The image generation model 60 generates a target image related to the inspection target based on the input corrected caption and division area information, and outputs the generated target image to the sentence generation model 70. The target image is an image segmented for each inspection part of the inspection target. When the inspection target is composed of a metal part and a resin part, the target image is composed of, for example, divided images obtained by dividing the metal part and the resin part.
[0023] When a target image is input, the sentence generation model 70 generates a document related to the inspection object of the target image and outputs the generated document to the evaluation function 23. The evaluation function 23 evaluates the input sentence based on the summary, and if it is not within an acceptable range, corrects the sentence and outputs the corrected sentence to the image generation model 60. The image generation model 60 generates the target image again based on the input corrected document, outputs the generated target image to the sentence generation model 70, and repeats the same process.
[0024] If the evaluation of the input sentence is within an acceptable range, the evaluation function 23 outputs the sentence to the sentence task generation model 81. The sentence task generation model 81 generates a plurality of tasks based on the input sentence, and outputs the generated plurality of tasks.
[0025] Next, each unit shown in FIG. 2 will be described in detail.
[0026] 3 is a diagram showing an example of the configuration of the first language generation model 40. The first language generation model 40 can be configured, for example, by BLIP, which is a VLP (Vision-Language Pre-training) framework, and includes an image encoder 41 (for example, ViT: Vision Transformer) and a text encoder 42 (for example, BERT: Bidirectional Encoder Representations from Transformers). The image encoder 41 extracts features from an input image and outputs the extracted features to the text encoder 42.
[0027] Fig. 4 is a diagram showing an example of an input image. As shown in the input image, the overall shape of the product to be inspected (referred to as product AA) is a circular object, and it can be seen that the product includes a disk-shaped metal part, two metal fittings attached to the outer periphery of the metal part in opposing relation to each other, and a resin part attached to the center of the disk. Note that the input image is merely an example, and is not limited to the example of Fig. 4.
[0028] Returning to FIG. 3, the text encoder 42 converts visual information into linguistic information based on the input features, and generates a caption for the input image.
[0029] Fig. 5 is a diagram showing an example of a caption for the input image shown in Fig. 4. As shown in Fig. 5, the caption is, for example, "The circular object is product AA. Product AA is composed of a disk-shaped metal part, two metal fittings attached to the outer periphery of the metal part, and a rectangular resin part attached to the center of the metal part. The metal part has a hole. The surface of the resin part is flat. ..." Note that the example of the caption is merely an example and is not limited to the example in Fig. 5.
[0030] As described above, the control unit 10 acquires an input image of the subject to be examined, and when an image is input, the control unit 10 can input the acquired input image into the first language generation model 40, which generates a caption for the image, to generate a caption for the input image.
[0031] 6 is a diagram showing an example of the configuration of the second language generation model 50. The second language generation model 50 can use Chat-GPT based on GPT-3, which employs a language model using a deep learning technique called Transformer, and includes an embedding unit 51, a decoding layer 52, and a conversion unit 53. When hearing content related to the test target is input, the embedding unit 51 breaks down the input hearing content (text) into tokens and converts them into multidimensional vectors (hearing vectors).
[0032] Fig. 7 is a diagram showing an example of the interview contents (text). The interview contents are, for example, the contents of a conversation about the product to be inspected between an inspector (customer) of a company that manufactures, inspects, and sells the product, and a person in charge of a company that manufactures and sells the inspection equipment used to inspect the product or that provides inspection technology support. By acquiring the interview contents, it is possible to find out the way in which the inspector (customer) thinks when conducting the inspection (for example, what kind of inspection is to be performed, the inspection procedure, the criteria for determining whether there are defects (inspection criteria, etc.), and points to note when conducting the inspection).
[0033] Returning to Fig. 6, the decoding layer 52 includes multiple Transformer Decoders, and calculates a score indicating the relationship between each token based on the input multidimensional vector (hearing vector), calculates a vector expressing a summary (summary vector) based on the calculated score, and outputs the calculated summary vector to the conversion unit 53. The conversion unit 53 generates a summary by converting the summary vector into words.
[0034] Fig. 8 is a diagram showing an example of a summary of the interview contents shown in Fig. 7. As shown in Fig. 8, the summary extracts information necessary for product inspection from the interview contents shown in Fig. 7, and deletes unnecessary information. This makes it possible to extract knowledge about the inspection that cannot be read from the input image alone, and to obtain information necessary for the inspection by complementing the visual information of the input image.
[0035] Fig. 9 is a diagram showing an example of a summary of the interview content regarding the inspection object shown in the input image of Fig. 4. For convenience, the interview content is omitted, but for an inspection object such as that shown in Fig. 4, a summary is generated such as "The metal part of product AA has four holes. It is prone to scratches near the four holes. Notch-like scratches are prone to occur near the outer periphery of the disc-shaped part. The surface of the resin part is prone to stains and scratches on the ceiling..." Note that the summary is only an example and is not limited to the example of Fig. 9.
[0036] As described above, the control unit 10 acquires hearing content related to the test subject, and when the hearing content (text) is input, the control unit 10 can input the acquired hearing content into the second language generation model 50, which generates a summary of the hearing content, to generate a summary.
[0037] The integration function 22 (control unit 10 ) corrects the caption generated by the first language generation model 40 based on the summary generated by the second language generation model 50 , and outputs the corrected caption to the image generation model 60 .
[0038] FIG. 10 is a diagram showing an example of a corrected caption obtained by correcting the caption in FIG. 5. The caption of the input image is obtained by converting visual information extracted from the input image into linguistic information, and expresses the visual information of the inspection target in the input image as language, so it includes information necessary for inspecting the inspection target and also includes information unnecessary for the inspection. For example, in the caption of the input image shown in FIG. 5, "the inspection target is product AA," "two metal fittings are attached to the periphery of the metal part of product AA," "there is a hole in the metal part," "the surface of the resin part is flat," and the like are known from the product specifications, and it is considered that it is not necessary to instruct the image generation model 60 (task generation device 100) as information that complements the input image. On the other hand, since the summary of the hearing contents includes useful information regarding the inspection of the inspection target, it is possible to generate a corrected caption by removing unnecessary information from the caption of the input image and adding useful information using information included in the summary. In the corrected caption shown in FIG. 10, unnecessary information is removed from the caption of the input image shown in FIG. 5, and necessary information is added from the summary shown in FIG. 9.
[0039] The divided area detection function 21 is a generative model that learns a style of dividing an area by adding an algorithm including a learning method for generating a new image, a function for extracting the contour and area of an image, a function for correcting a part of an image, and a function for correcting the contour and area to a diffusion model that generates an image. When a normal product image is input, the divided area detection function 21 generates divided area information and outputs the generated divided area information to the image generation model 60. The divided area information includes information necessary for dividing an area. For example, it includes information for dividing an inspection object by appearance features (e.g., edges, color changes, lines, etc.), information for dividing an inspection object by grouping divided areas according to inspection convenience or function (e.g., surface roughness, the presence or absence of additional processing in a post-process, or differences, etc.) or lighting convenience, and information for dividing an inspection object according to human sensory requirements (e.g., vague specification of how many millimeters from an edge).
[0040] 11 is a diagram showing an example of the configuration of an image generation model 60. The image generation model 60 can be configured, for example, with Text2Image, and includes a text encoder 61, a diffusion model 62, an image decoder 63, and an image processing unit 64. When a corrected caption (text) is input, the text encoder 61 converts each word into a word vector, and then uses a Transformer to extract a feature vector (text feature vector) indicating the meaning of the text. The text encoder 61 outputs the extracted text feature vector to the diffusion model 62.
[0041] The diffusion model 62 converts the input text feature vector into an image feature vector, and outputs the converted image feature vector to the image decoder 63 .
[0042] When the image feature vector is input, the image decoder 63 converts the image feature vector into an image and outputs the converted image to the image processing unit 64. The image decoder 63 can generate a provisional image of the target image obtained by segmenting the inspection target.
[0043] The image processing unit 64 generates a target image by modifying the regions to be divided and the contours based on the divided region information for the provisional image of the target image output by the image decoder 63. The image generation model 60 outputs the generated target image to the sentence generation model 70.
[0044] In this case, the image decoder 63 generates a target image by segmenting the inspection target, and outputs the generated target image to the sentence generation model 70.
[0045] Fig. 12 is a diagram showing an example of a target image. The target image shown in Fig. 12 corresponds to the inspection target of the input image shown in Fig. 4. As shown in Fig. 12, the target image is divided into three types of segments, for example, a segment S1 indicating a metal part, a segment S2 indicating a resin part, and a segment S3 indicating a metal fitting. Each segment is also called a divided image. The target image includes divided images segmented for each part to be inspected.
[0046] As described above, when the control unit 10 inputs text, the first language generation model 40 inputs a generated caption to the image generation model 60, which generates an image that conforms to the text, to generate a target image that conforms to the caption, and the generated target image can be obtained.
[0047] Furthermore, the control unit 10 may input the caption corrected by the integration function 22 to the image generation model 60 to generate a target image.
[0048] 13 is a diagram showing an example of the configuration of a sentence generation model 70. The sentence generation model 70 can be configured, for example, by BLIP or the like, and includes an image encoder 71 (for example, ViT: Vision Transformer) and a text encoder 72 (for example, BERT: Bidirectional Encoder Representations from Transformers). The image encoder 71 extracts features from a target image and outputs the extracted features to the text encoder 72.
[0049] The text encoder 72 converts the visual information into linguistic information based on the input feature amount, and generates a sentence related to the inspection target of the target image. The sentence generation model 70 outputs the generated sentence to the evaluation function 23. Note that, when the generated sentence does not need to be evaluated, the sentence generation model 70 may output the generated sentence to the sentence task generation model 81.
[0050] Fig. 14 is a diagram showing an example of text (before evaluation) regarding the inspection target. In the example of Fig. 14, the text regarding the inspection target is as follows: "The inspection target is composed of three parts: a metal part, a resin part, and a metal fitting. The metal part is disk-shaped and has four holes. The areas around the four holes are prone to scratches. Notch-shaped scratches are prone to form near the outer periphery of the metal part. The product model number is engraved on the surface of the metal part. A rectangular resin part is attached to the center of the disk-shaped metal part. The surface of the resin part is flat and has a hardness value between XX and △△, so small scratches are prone to form on the surface. The metal fittings are attached to two points on the outer periphery of the metal part. The metal fittings are not subject to inspection..."
[0051] As described above, when the control unit 10 inputs an image of an inspection target, it can input the acquired target image to the sentence generation model 70, which generates sentences related to the inspection target, to generate sentences related to the inspection target of the target image.
[0052] The evaluation function 23 (control unit 10) acquires a summary of the hearing contents related to the test subject, evaluates the sentence generated by the sentence generation model 70 based on the acquired summary, and corrects the sentence generated by the sentence generation model 70 based on the evaluation result. Specifically, the evaluation function 23 calculates the similarity between the sentence generated by the sentence generation model 70 and the summary, determines whether the calculated similarity is within an allowable range, and if it is not within the allowable range, corrects the sentence generated by the sentence generation model 70, outputs the corrected sentence to the image generation model 60, and repeats the processing by the image generation model 60. For example, a method such as a rule-based method may be used to correct the sentence. On the other hand, if the calculated similarity is within an allowable range, the sentence generated by the sentence generation model 70 is output to the sentence task generation model 81.
[0053] The sentences generated by the sentence generation model 70 can be evaluated by the following procedure. That is, (1) morphological analysis is performed on each of the sentences and summaries generated by the sentence generation model 70 to break them down into words. (2) Words that appear in the sentences and summaries are arranged, and words used in the sentences and summaries are assigned the value 1, and words not used are assigned the value 0, thereby vectorizing the sentences and summaries. (3) The cosine similarity between the vectors of the sentences and summaries is calculated, and if the calculated cosine similarity is equal to or greater than a predetermined threshold, it can be determined that it is within an acceptable range. Note that the calculation of similarity is not limited to cosine similarity.
[0054] Fig. 15 is a diagram showing an example of a sentence (after evaluation) regarding the inspection object. In the example of Fig. 15, the sentence regarding the inspection object is as follows: "The inspection object is a disk-shaped metal part and a resin part. The product model number is engraved on the metal part, and scratches are likely to occur around the four holes, and notch-shaped scratches are likely to occur near the outer periphery. Small scratches are likely to occur on the surface of the resin part..."
[0055] The text task generation model 81 can be configured, for example, with Graph-GPT (GPT-3) or the like, and generates multiple tasks based on input text (text related to the inspection target) by converting unstructured natural language into a knowledge graph. The multiple tasks can be generated in a directed graph or JSON (JavaScript Object Notation) format.
[0056] Fig. 16 is a diagram showing an example of a process of converting a sentence related to an inspection target into a directed graph by the sentence task generation model 81. Fig. 16 shows five steps, No. 1 to No. 5. In the first step, the phrase "inspection target" in the sentence is converted into a directed graph of three tasks, "input image" -> "target detection" -> "detection score map correction."
[0057] In the second step, the phrase "The inspection objects are...metal parts and plastic parts" in the text is converted into a directed graph of two tasks, "detection score map correction" → "type classification," following the "detection score map correction" in the first step.
[0058] In the third step, the sentence “The product model number is engraved on the metal part…” is converted into a directed graph of three tasks: “Type classification” → “Character extraction” → “Text detection”, following the “Type classification” in the second step.
[0059] In the fourth step, the sentence “Metal parts are…easily scratched…and prone to notch-like scratches” is converted into a directed graph of three tasks: “type classification” → “image processing” → “defect detection,” following the “type classification” in the second step.
[0060] In the fifth step, the sentence "The surface of plastic parts is prone to small scratches" is converted into a directed graph of three tasks: "type classification" → "image processing" → "defect detection," following the "type classification" in the second step.
[0061] Fig. 17 is a diagram showing an example of multiple tasks generated by the text task generation model 81 (task generation model 80). The multiple tasks shown in Fig. 17 are a compilation of the directed graph shown in Fig. 16. In Fig. 17, the tasks are "input image", "object detection" M1, "detection score map correction", "type classification" M2, "character extraction" M3, "text detection" M4, "image processing", "defect detection" M5, "image processing", and "defect detection" M6. The symbols M1 to M6 represent different learning models that perform processing specialized for the respective tasks.
[0062] Fig. 18 is a diagram showing an example of the function of a learning model dedicated to each task. The task "object detection" M1 is processed by a dedicated learning model that performs segmentation, and the learning model detects the object (workpiece) to be inspected.
[0063] The task "type classification" M2 is processed by a learning model dedicated to classification, and the learning model classifies the inspection objects into metal parts, resin parts, and metal fittings.
[0064] The task “character extraction” M3 is processed by a dedicated learning model for segmentation, which extracts character regions within the inspection object.
[0065] The task "Text detection" M4 is processed by a learning model dedicated to character recognition, which detects text by recognizing characters within character regions.
[0066] The task “defect detection” M5 is processed by a dedicated learning model for segmentation, which detects defects in metal parts.
[0067] The task “defect detection” M6 is processed by a dedicated learning model that performs segmentation, and the learning model detects defective areas in plastic parts.
[0068] The task "detection score map correction" adjusts the area of the object detected in task "object detection" M1 by deleting part of the area or by connecting separated areas. The task "image processing" performs preprocessing to improve the accuracy of defect detection according to the attributes of the inspection area (in this case, metal parts and plastic parts).
[0069] As described above, the control unit 10 can input the sentences generated by the sentence generation model 70 to the sentence task generation model 81 to generate a plurality of tasks for inspecting the inspection target. The control unit 10 may also input the sentences corrected by the evaluation function 23 to the sentence task generation model 81 to generate a plurality of tasks for inspecting the inspection target.
[0070] Each of the multiple tasks can be performed using one of an object detection model that detects an inspection object, a classification model that classifies an inspection portion of the inspection object, a defect candidate detection model that detects defect candidates in the inspection portion, or a defect determination model that determines defects in the defect candidates.
[0071] As described above, the text task generation model 81 (task generation model 80) can generate one or more tasks for each divided image segmented for each inspection target part. The text task generation model 81 (task generation model 80) can generate multiple tasks including GPT and with a specified processing order. The text task generation model 81 (task generation model 80) can generate multiple tasks displayed in a graph.
[0072] As described above, according to this embodiment, multiple tasks are generated for inspecting an object to be inspected, so there is no need to create programming to construct an inspection algorithm, and each task is realized by a dedicated learning model that performs processing according to the inspection part of the object to be inspected, making it possible to inspect objects for which the inspection criteria differ depending on the inspection part.
[0073] In addition, in general image inspection, a learning model is often used to estimate an area and determine whether an object to be inspected is good or bad by estimating the class of the estimated area, but such learning models are processed in the same way for all objects. However, in actual appearance inspection, there is a large variation in aspect ratio, size, occurrence frequency, etc. for each class (inspection target part), and each class often has different individual adjustments and inspection standards. According to this embodiment, the inspection target is not inspected using one learning model, but multiple tasks are generated that determine processing by dedicated or individual learning models optimized for each inspection part according to the inspection part of the inspection target, so that inspection of an inspection target with various inspection standards can be performed.
[0074] 19 is a diagram showing an example of processing of the defect image generation model 90. The defect image generation model 90 includes a first defect image generation model 91 and a second defect image generation model 92. The first defect image generation model 91 and the second defect image generation model 92 are also collectively referred to as the defect image generation model 90. The defect image generation model 90 can be configured as a multimodal generation model such as SceneComposer, and when a target image relating to an inspection target is input, a defect image depicting a defect (e.g., a scratch, etc.) can be generated. When the target image is input, setting conditions are also input, so that a defect image according to the setting conditions is generated.
[0075] The set conditions can be expressed in text. The set conditions include, for example, the inspection specifications for the inspection target, the contents of interviews regarding the inspection target, etc. The inspection specifications can be set, for example, as "strict inspection within XX mm from the outer circumference of the part to be inspected", "strict inspection of the surface of resin parts", "strict inspection of the surroundings of holes in metal parts", etc. Note that the set conditions are not limited to these.
[0076] In the example of FIG. 19, a first defect image generation model 91 specialized for metal parts is used for an image of a metal part to be inspected (metal part image). Also, a second defect image generation model 92 specialized for resin parts is used for an image of a resin part to be inspected (resin part image). In this way, it is possible to use an optimal defect image generation model depending on the inspection part of the inspection object. Note that the inspection part is not limited to a metal part or a resin part.
[0077] As described above, when the control unit 10 receives an image relating to an inspection object, it can input the acquired target image into the defect image generation model 90, which generates a defect image showing defects in the inspection area, to generate a defect image of the inspection area of the target image.
[0078] In addition, the control unit 10 can input at least one of the segmented inspection area of the acquired target image, the hearing contents regarding the inspection object, and the inspection specifications regarding the inspection object into the defect image generation model 90 to generate a defect image.
[0079] The defect images generated by the defect image generation model 90 can be collected, and the collected defect images can be used as teacher data for learning or relearning the learning model (defect judgment model) used when executing the tasks "defect detection" M5 and M6 shown in Figure 17.
[0080] That is, the control unit 10 can learn or re-learn the defect judgment model for a task executed using the defect judgment model for determining defects in an inspection area by using the generated defect image as training data, thereby improving the accuracy of defect detection.
[0081] Fig. 20 is a diagram showing a second example of a processing scheme of the task generation device 100. The difference from the first example shown in Fig. 2 is that the sentence generation model 70, the evaluation function 23, and the sentence task generation model 81 of the first example are not included. In the second example, the target image generated by the image generation model 60 is output to the task generation model 80. The other points are the same as those of the first example, and therefore the description will be omitted.
[0082] The task generation model 80 is composed of, for example, GPT+Graph-GPT, and when a target image is input, it generates and outputs multiple tasks.
[0083] As described above, the control unit 10 acquires an object image relating to an inspection object, and when the control unit 10 inputs the image of the inspection object, the control unit 10 can input the acquired object image into the task generation model 80, which generates a plurality of tasks for inspecting the inspection object, and generate a plurality of tasks for inspecting the inspection object in the object image.
[0084] As described above, multiple tasks are generated for inspecting the object to be inspected, so there is no need to create programming to construct an inspection algorithm, and each task is realized by a dedicated learning model that performs processing according to the inspection part of the object to be inspected, making it possible to inspect objects where the inspection criteria differ depending on the inspection part.
[0085] 21 is a diagram showing a third example of a processing scheme of the task generation device 100. The third example includes a first language generation model 40, a second language generation model 50, an integration function 22, and a task generation model 80. The first language generation model 40, the second language generation model 50, and the integration function 22 are similar to those in the first example, and therefore their explanation will be omitted.
[0086] The task generation model 80 can be configured, for example, as a multimodal GPT-4 that can input images and text. When an input image and a corrected caption generated by the integration function 22 are input, the task generation model 80 can generate and output multiple tasks.
[0087] That is, the control unit 10 acquires the caption generated by the first language generation model 40 or the corrected caption corrected by the integration function 22, and an input image related to the subject of inspection, and when the control unit 10 inputs the caption or the corrected caption, and the input image, it can generate multiple tasks by inputting the acquired caption or the corrected caption, and the input image into a task generation model 80 that generates multiple tasks.
[0088] In this way, multiple tasks are generated for inspecting the object to be inspected, eliminating the need to create programming to construct an inspection algorithm. Furthermore, each task is realized by a dedicated learning model that performs processing according to the inspection part of the object to be inspected, making it possible to inspect objects for which the inspection criteria differ depending on the inspection part.
[0089] Next, a display screen displayed on the task generation device 100 (the display unit 13 or an external display device) will be described.
[0090] FIG. 22 is a diagram showing an example of a task creation screen 210. The user can specify a desired project name, and by selecting a data set name, thumbnail images of the input image and the normal product image are displayed. The user can further select the hearing contents (file) related to the inspection target. Alternatively, the user may input the text of the hearing contents in the comment field 211. The user can specify a desired group name. For example, a different group name can be specified for each product lot. By operating the "Create" icon, processing is performed in each section within the task generation device 100, and multiple tasks are generated. In addition, by operating the "Cancel" icon, the selected or input contents can be canceled.
[0091] FIG. 23 is a diagram showing an example of an inspection setting screen 220. A user can specify a desired project name and select a desired task from a list of multiple tasks generated. In the example of FIG. 23, a task: XX defect detection 221 is selected. A user can set parameters for when the selected task is executed in a setting field 222. For example, as shown in FIG. 23, when a task is a task related to inspection, the user can specifically set the inspection contents. The inspection contents include, for example, the type of scratch, the length of the scratch, the area of the scratch, the number of scratches, the total area of the scratches, and the like. By setting the inspection contents, the conditions for determining the presence or absence of defects can be set. The inspection contents are set by operating the "setting" icon. In addition, the selected or input contents can be canceled by operating the "cancel" icon.
[0092] FIG. 24 is a diagram showing an example of a task annotation screen 230. In the task flow field, a plurality of generated tasks are displayed in a graph. The user can select a task to be annotated. The user can select a desired task by moving the cursor 231 to the desired task (for example, the task "defect detection": the task with the pattern in the figure). In the annotation content field, an image of the inspection target (target image or input image) is displayed. The user can edit the image of the inspection target by selecting a desired tool from the editing tools 233. In the example shown in the figure, an annotation is added to a part of the outer periphery of the inspection target using the tool 232 (the part with the pattern in the figure). In addition, the user can input instructions regarding the annotation in the input field 234. In the example shown in the figure, the instruction "add to defect detection area" is input. The annotation is executed by operating the "execute" icon. In addition, the editing contents or the input contents can be canceled by operating the "cancel" icon.
[0093] The learning model performing the selected task can detect defects using the regions added by the annotation as defect detection regions.
[0094] FIG. 25 is a diagram showing an example of a normal product image annotation screen 240. The user can specify a desired project name, and by selecting a data set name, the original image of the inspection target is displayed, and an edited image field for displaying an edited image is displayed. Before editing, the original image is displayed in the edited image field. The user can edit the original image by selecting a desired tool from the editing tools 242. In the example shown, the tool 241 is used to edit the area of the inspection part in the original image (the patterned part in the figure). Editing is executed by operating the "Execute" icon. Also, the edit contents can be canceled by operating the "Cancel" icon.
[0095] The divided area detection function 21 can generate divided area information based on the edited image.
[0096] 26 is a diagram showing an example of a target image annotation screen (before annotation) 250. The target image annotation screen 250 graphically displays a target image and a plurality of tasks generated based on the target image. The target image before annotation is assumed to be segmented (divided) into two regions indicated by reference numerals 251 and 252. The directed graph of the plurality of tasks branches into two midway.
[0097] Fig. 27 is a diagram showing an example of a target image annotation screen (after annotation) 260. As shown in Fig. 27, when two segments 251 and 252 of the target image are corrected to one segment 261 by annotation, the directed graph of multiple tasks is automatically corrected and no more branches are present. The number and types of tasks are also appropriately changed.
[0098] In this way, the control unit 10 can accept editing operations for the target image generated by the image generation model 60 and change multiple tasks according to the accepted editing operations. This allows the user to change multiple tasks for inspecting the inspection target simply by editing the target image related to the inspection target, eliminating the need for specialized engineers with programming skills and image processing skills, and allowing the user to change the inspection procedure and inspection contents without coding, making it easy to inspect various inspection sites.
[0099] FIG. 28 is a diagram showing an example of a task flow selection screen 270. The task flow selection screen 270 displays the task flows and workloads of candidates 1, 2, and 3. The user can select a desired task flow from among multiple candidates by selecting one of selection boxes 271, 272, and 273 for selecting candidates 1, 2, and 3, respectively. In the example of FIG. 28, candidate 1 is selected. The selection is executed by operating the "Execute" icon. In addition, the selection in the selection box can be canceled by operating the "Cancel" icon.
[0100] In this way, the control unit 10 can display multiple candidates for multiple tasks generated by the task generation model 80 and accept the selection of any candidate from the multiple displayed candidates. This allows the user to select an appropriate task flow even when the same inspection target has different manufacturing methods or uses, or when customer requirements are different.
[0101] 29 is a diagram showing an example of a processing procedure of the task generation device 100. For convenience, the processing will be described as being performed by the control unit 10. The control unit 10 acquires an input image of an object to be inspected (S11), and acquires a normal product image of a normal part (S12). The control unit 10 acquires hearing contents regarding the object to be inspected (S13).
[0102] The control unit 10 generates segmented region information based on the normal product image (S14). The control unit 10 inputs the input image to the first language generation model 40 to generate a caption for the input image (S15), and inputs the hearing content to the second language generation model 50 to generate a summary of the hearing content (S16).
[0103] The control unit 10 corrects the caption based on the generated summary (S17). For the correction of the caption, see Fig. 5 and Fig. 10. The control unit 10 inputs the corrected caption and the divided region information to the image generation model 60, and generates a target image (S18).
[0104] The control unit 10 inputs the target image into the sentence generation model 70 to generate sentences related to the test target (S19), and evaluates the generated sentences based on the summary of the hearing contents (S20). The evaluation can use the cosine similarity between the sentences and the summary. The control unit 10 determines whether the evaluation result is within an acceptable range (S21), and if it is not within the acceptable range (NO in S21), inputs the corrected sentences into the image generation model 60 to generate the target image (S22), and continues the processing from step S19 onwards.
[0105] If the evaluation result is within the allowable range (YES in S21), the control unit 10 inputs the generated sentence into the sentence task generation model 81 to generate a plurality of tasks (S23), and ends the process.
[0106] According to this embodiment, it is possible to set the examination appropriately and in detail according to the examination items specified in all or part of the multiple tasks. In addition, since a task corresponds to one learning model, the entire examination of the examination object can be performed using multiple dedicated learning models, and the function and role of each learning model are clear, the grounds and reasons for the examination results become relatively clear, and the examination criteria for the examination area can be quantified. In addition, since processing can be performed at the required resolution for each task, there is no need to use a uniformly high resolution, and processing can be performed at high speed.
[0107] (Additional Note 1) The computer program causes a computer to execute a process of acquiring an object image relating to an inspection object, and when the image of the inspection object is input, inputting the acquired object image into a task generation model which generates a plurality of tasks for inspecting the inspection object, and generating a plurality of tasks for inspecting the inspection object in the object image.
[0108] (Appendix 2) The computer program in Appendix 1 causes a computer to execute a process in which the task generation model includes a sentence task generation model that, when sentences related to an inspection object are input, generates a plurality of tasks for inspecting the inspection object, and, when an image of the inspection object is input, inputs an acquired target image into the sentence generation model that generates sentences related to the inspection object to generate sentences related to the inspection object in the target image, and inputs the generated sentences into the sentence task generation model to generate a plurality of tasks for inspecting the inspection object.
[0109] (Appendix 3) The computer program in Appendix 2 causes a computer to execute a process of obtaining a summary of the contents of a hearing regarding a test subject, evaluating the sentences generated by the sentence generation model based on the obtained summary, correcting the sentences generated by the sentence generation model based on a result of the evaluation, and inputting the corrected sentences into the sentence task generation model to generate a plurality of tasks for testing the test subject.
[0110] (Appendix 4) In the computer program of any one of Appendices 1 to 3, each of the plurality of tasks is executed using any one of an object detection model that detects an inspection object, a classification model that classifies an inspection portion of the inspection object, a defect candidate detection model that detects defect candidates in the inspection portion, or a defect determination model that determines defects in the defect candidates.
[0111] (Appendix 5) A computer program, in any one of Appendices 1 to 4, causes a computer to execute a process of acquiring an input image of an object to be inspected, and, when an image is input, inputting the acquired input image into a first language generation model that generates a caption for the image to generate a caption for the input image, and, when text is input, inputting the generated caption into an image generation model that generates an image in accordance with the text to generate an object image of the object to be inspected in accordance with the caption, and acquiring the generated object image.
[0112] (Appendix 6) The computer program in Appendix 5 causes a computer to execute the following processes: acquire hearing content related to a test subject; when the hearing content is input, input the acquired hearing content into a second language generation model that generates a summary of the hearing content to generate a summary; based on the generated summary, modify a caption generated by the first language generation model; and input the modified caption into the image generation model to generate the target image.
[0113] (Supplementary Note 7) In the computer program according to any one of Supplementary Note 1 to Supplementary Note 6, the target image includes divided images segmented for each inspection target area.
[0114] (Supplementary Note 8) In the computer program according to any one of Supplementary Note 1 to Supplementary Note 7, the task generation model generates one or more tasks for each divided image segmented for each inspection target area.
[0115] (Supplementary Note 9) In the computer program according to any one of Supplementary Note 1 to Supplementary Note 8, the task generation model includes a GPT, and generates a plurality of tasks with a specified processing order.
[0116] (Supplementary Note 10) In the computer program according to any one of Supplementary Note 1 to Supplementary Note 9, the task generation model generates a plurality of tasks displayed in a graph.
[0117] (Supplementary Note 11) The computer program in Supplementary Note 5 causes a computer to execute a process of accepting an editing operation on a target image generated by the image generation model, and changing the plurality of tasks in accordance with the accepted editing operation.
[0118] (Appendix 12) A computer program in any one of Appendices 1 to 11, which causes a computer to execute a process of displaying multiple candidates for multiple tasks generated by the task generation model, and accepting selection of any candidate from the displayed multiple candidates.
[0119] (Appendix 13) The computer program according to any one of Appendices 1 to 12 causes a computer to execute a process in which, when an image relating to an inspection object is input, the acquired target image is input into a defect image generation model that generates a defect image showing a defect in the inspection area, and a defect image of the inspection area of the target image is generated.
[0120] (Appendix 14) The computer program in appendix 13 causes a computer to execute a process of inputting at least one of the segmented inspection area of the acquired target image, the hearing contents regarding the inspection object, and the inspection specifications regarding the inspection object into the defect image generation model to generate the defect image.
[0121] (Addendum 15) The computer program according to Addendum 13 or Addendum 14 causes a computer to execute a process in which some of the tasks are executed using a defect determination model that determines defects in an inspection area, and the defect determination model is trained or retrained using a generated defect image as training data.
[0122] (Addendum 16) The task generation device includes a control unit, which acquires an object image relating to an inspection object, and when the control unit acquires the image of the inspection object, inputs the acquired object image into a task generation model which generates a plurality of tasks for inspecting the inspection object, thereby generating a plurality of tasks for inspecting the inspection object in the object image.
[0123] (Appendix 17) A task generation method acquires an object image relating to an inspection object, and when the image of the inspection object is input, the acquired object image is input into a task generation model which generates a plurality of tasks for inspecting the inspection object, and generates a plurality of tasks for inspecting the inspection object in the object image.
[0124] The matters described in each embodiment can be combined with each other. In addition, the independent claims and dependent claims described in the claims can be combined with each other in any and all combinations regardless of the citation format. Furthermore, the claims use a format in which a claim cites two or more other claims (multiple claim format), but this is not limited to this. A multiple claim (multi-multi claim) that cites at least one multiple claim may also be used. [Explanation of symbols]
[0125] 10 Control section 11 Communications Department 12. Memory 13 Display section 14 Control section 15 Storage section 16 Computer Programs 20 Processing section 21 Split area detection function 22 Integration Features 23 Evaluation Function 30 Learning Model Section 40 First Language Generation Model 41 Image Encoder 42 Text Encoder 50 Second Language Generation Model 51 Embedded part 52 Decoding Layer 521 Transformer Decoder 53 Conversion section 60 Image Generation Model 61 Text Encoder 62 Diffusion Model 63 Image Decoder 64 Image Processing Unit 70 Text Generation Model 71 Image Encoder 72 Text Encoder 80 Task Generation Model 81 Text Task Generation Model 90 Defect Image Generation Model 91 First defect image generation model 92 Second defect image generation model
Claims
1. acquiring an object image relating to an object of inspection; a task generation model that generates a plurality of tasks for inspecting an inspection object in a case where an image of the inspection object is input, inputting the acquired target image into the task generation model that generates a plurality of tasks for inspecting the inspection object; A computer program that causes a computer to carry out processing.
2. The task generation model is a text task generation model that generates a plurality of tasks for inspecting a text related to an inspection object when the text is input; When an image of an inspection target is input, the acquired target image is input to a sentence generation model that generates sentences related to the inspection target, thereby generating sentences related to the inspection target of the target image; inputting the generated sentences into the sentence task generation model to generate a plurality of tasks for inspecting the inspection object; 2. A computer program product according to claim 1, which causes a computer to carry out a process.
3. Obtain a summary of the hearings regarding the subject of the inspection; Evaluating the sentences generated by the sentence generation model based on the obtained summary; Modifying the sentences generated by the sentence generation model based on the evaluation results; inputting the modified sentence into the sentence task generation model to generate a plurality of tasks for testing the test object; 3. A computer program product according to claim 2, which causes a computer to carry out a process.
4. each of the plurality of tasks is executed using one of an object detection model for detecting an inspection object, a classification model for classifying an inspection portion of the inspection object, a defect candidate detection model for detecting a defect candidate in the inspection portion, and a defect determination model for determining a defect in the defect candidate.
2. The computer program product of claim 1.
5. Acquire an input image of the inspection object; When an image is input, the acquired input image is input to a first language generation model that generates a caption for the image, thereby generating a caption for the input image; When text is input, the generated caption is input to an image generation model that generates an image according to the text, and an object image of the inspection object according to the caption is generated; Obtain the generated target image; A computer program product according to any one of claims 1 to 4, which causes a computer to execute a process.
6. Obtain the contents of the hearing regarding the subject of the inspection, When the hearing contents are input, the acquired hearing contents are input to a second language generation model that generates a summary of the hearing contents to generate a summary; modifying the caption generated by the first language generation model based on the generated summary; inputting the modified caption into the image generation model to generate the target image; 6. A computer program product according to claim 5, which causes a computer to carry out a process.
7. The target image includes divided images segmented for each inspection target region. A computer program according to any one of claims 1 to 4.
8. The task generation model is generating one or more tasks for each segmented image for each inspection target area; A computer program according to any one of claims 1 to 4.
9. The task generation model is Generate multiple tasks including GPT and with a specific processing order. A computer program according to any one of claims 1 to 4.
10. The task generation model is Generate multiple tasks with a graphical representation, A computer program according to any one of claims 1 to 4.
11. Accepting an editing operation on a target image generated by the image generation model; changing the plurality of tasks in response to the received editing operation; 6. A computer program product according to claim 5, which causes a computer to carry out a process.
12. Displaying a plurality of candidates for a plurality of tasks to be generated by the task generation model; Accept the selection of any candidate from among the multiple candidates displayed, 5. A computer program product according to claim 1, which causes a computer to carry out a process.
13. When an image relating to an inspection object is input, the acquired object image is input to a defect image generation model that generates a defect image showing a defect in the inspection portion, thereby generating a defect image of the inspection portion of the object image.
5. A computer program product according to claim 1, which causes a computer to carry out a process.
14. inputting at least one of the segmented inspection portion of the acquired target image, the hearing contents related to the inspection object, and the inspection specifications related to the inspection object into the defect image generation model to generate the defect image; 14. A computer program product according to claim 13, which causes a computer to carry out a process.
15. A portion of the plurality of tasks is executed using a defect determination model for determining defects in the inspection portion; learning or relearning the defect determination model using the generated defect image as training data; 14. A computer program product according to claim 13, which causes a computer to carry out a process.
16. A control unit is provided, The control unit is acquiring an object image relating to an object of inspection; a task generation model that generates a plurality of tasks for inspecting an inspection object in a case where an image of the inspection object is input, inputting the acquired target image into the task generation model that generates a plurality of tasks for inspecting the inspection object; Task generation device.
17. acquiring an object image relating to an object of inspection; a task generation model that generates a plurality of tasks for inspecting an inspection object in a case where an image of the inspection object is input, inputting the acquired target image into the task generation model that generates a plurality of tasks for inspecting the inspection object; How to create a task.
Citation Information
Patent Citations
Image inspection device and method for setting image inspection device
JP2020187070A
Cited By
Information processing device, information processing method, and program
JP2025153560A
Information processing device, information processing method, and program
JP2025153946A
Information processing device, information processing system, information processing method, and program
JP2026144955A
Information processing device, information processing method, and program
JP7670896B1
Information processing device, information processing method, and program
JP7762249B2
