Machine learning system, learning data collection method, and storage medium

Abstract

A machine learning system includes a photographing unit that photographs a product to obtain a product image; a preprocessing unit that generates an inspection target portion image by cutting out an image of an inspection target portion according to a setting file, and stores the generated inspection target portion image in an image storage unit; and an inspection processing unit that judges the inspection target portion image of an object to be judged for good or not, and performs a good or not judgment processing. An object to be judged for good or not is indicated as an object to be judged for good or not by production instruction information. An inspection target portion image related to a learning object that is specified as not an object to be judged for good or not by the production instruction information, among the inspection target portion images stored in the image storage unit, is accumulated in the image storage unit as learning data.

Description

Technical Field

[0001] This invention relates to a machine learning system, a learning data collection method, and a storage medium. For example, it relates to a machine learning system, a learning data collection method, and a storage medium for generating a learning model that is used in artificial intelligence to perform shape recognition of an object being inspected using images captured on the object. Background Technology

[0002] In automobile production lines, image-based inspection is sometimes used in the inspection process where assembled products are checked according to specifications. In recent years, a technique has been proposed that uses artificial intelligence to determine the quality of an object under inspection based on an image containing that object. An example of such an identification and inspection technique is disclosed in International Publication No. 2019 / 230356.

[0003] The learning apparatus described in International Publication No. 2019 / 0230356 includes: a camera that captures image data of product samples; a physical attribute information acquisition unit that acquires physical attribute information of the samples; and a computing unit that generates a learning model. The computing unit is configured to determine the category of the sample based on rule information that associates physical attribute information with categories, generate supervised data by associating the determined categories with the image data, and generate a learning model using machine learning with the supervised data. The learning model outputs the category of the sample as input to the image data of the sample. Summary of the Invention

[0004] However, in the technology described in International Publication No. 2019 / 0230356, in order to obtain physical property information, it is necessary to set up other sensors besides the camera that acquires the image separately, which complicates or increases the cost of the system and further reduces its versatility.

[0005] This invention was made to solve such a problem, and its purpose is to simplify the system structure.

[0006] One aspect of the present invention is a machine learning system comprising: an imaging unit that captures images of a product; a preprocessing unit that, based on the product image, generates an inspection object image by cutting out an image of the inspection object part according to a setting file indicating the position and range of the inspection object part of the product, and stores the generated inspection object image in an image storage unit; and an inspection processing unit that performs good / bad judgment processing using artificial intelligence on the inspection object image of the inspection object, wherein the good / bad judgment object is represented as a good / bad judgment object by production instruction information describing the specifications of the product, and, regarding the inspection object image stored in the image storage unit, if it is an inspection object image related to a product that is designated as not a good / bad judgment object by the production instruction information, it is accumulated in the image storage unit as learning data for the learning model applied in the artificial intelligence.

[0007] One aspect of the present invention is a learning data collection method using an inspection device that performs product quality assessment in a production line. The learning data is used to generate a learning model applied in artificial intelligence. The learning data collection method performs the following processes: image acquisition, capturing images of products flowing in the production line to obtain product images; preprocessing, generating inspection object images by cutting out images of the inspection object parts according to a setting file indicating the position and range of the inspection object parts of the product, and storing the generated inspection object image in an image storage unit; inspection processing, performing quality assessment processing using artificial intelligence on the inspection object image of the quality assessment object, wherein the quality assessment object is represented as a quality assessment object by production instruction information describing the product specifications; and accumulating inspection object images stored in the image storage unit as learning data for the learning model when they are related to products that are designated as not quality assessment objects by the production instruction information.

[0008] One aspect of the present invention is a storage medium storing a learning data collection program. This program is executed by a computing unit within an inspection device to collect learning data. The inspection device performs product quality judgment processing on a production line. The learning data is used to generate a learning model for artificial intelligence applications using the inspection device. The learning data collection program includes: image processing, capturing images of products flowing on the production line to obtain product images; and preprocessing, generating inspection object images by cutting out images of the inspection object parts based on a setting file indicating the position and range of the inspection object parts of the product. The generated images of the inspection targets are stored in the image storage unit. The inspection process involves performing a good / bad judgment process using artificial intelligence on the images of the inspection targets, where the good / bad judgment targets are represented by production instruction information describing the product specifications. Regarding the inspection target images stored in the image storage unit, if they are images related to a product that is designated as not a good / bad judgment target by the production instruction information, they are accumulated in the image storage unit as learning data for the learning model.

[0009] In the machine learning system, learning data collection method, and storage medium of the present invention, the collection of learning data for generating a learning model is carried out in the system that performs the inspection processing.

[0010] According to the present invention, the system structure can be simplified in the machine learning system, learning data collection method and storage medium of the present invention. Attached Figure Description

[0011] Hereinafter, with reference to the accompanying drawings, the features, advantages, and technical and industrial significance of exemplary embodiments of the present invention will be described, in which the same symbols denote the same elements.

[0012] Figure 1 This is a schematic diagram of the machine learning system in Implementation Method 1.

[0013] Figure 2 This is an example of production instruction information used in the machine learning system of Implementation 1.

[0014] Figure 3 This is an example of the description of the configuration file used in the machine learning system of Implementation 1.

[0015] Figure 4 This is a diagram illustrating the preprocessing of the machine learning system in Implementation Method 1.

[0016] Figure 5This is a flowchart illustrating the operation of the machine learning system in Implementation Method 1. Detailed Implementation

[0017] For clarity of explanation, the following descriptions and figures have been appropriately omitted and simplified. Furthermore, as functional blocks performing various processes, the elements described in the figures can be implemented in hardware as a CPU (Central Processing Unit), memory, and other circuits, and in software as a program loaded into memory. Therefore, those skilled in the art will understand that these functional blocks can be implemented in various forms, either solely in hardware, solely in software, or through a combination thereof, and are not limited to any one of them. Moreover, in the figures, the same symbols are used to denote the same elements, and repeated descriptions are omitted as necessary.

[0018] Furthermore, the aforementioned program includes a set of instructions (or software code) that, when read into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a physical storage medium. By way of example, and not limitation, computer-readable media or physical storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other storage technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, cassette tape, magnetic tape, disk storage, or other magnetic storage devices. The program may be transmitted on a temporary computer-readable medium or a communication medium. By way of example, and not limitation, temporary computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagation signals.

[0019] Implementation Method 1

[0020] first, Figure 1 This is a schematic diagram illustrating the machine learning system 1 according to implementation method 1. Figure 1 As shown, the machine learning system 1 of Embodiment 1 includes an inspection device 10 that uses artificial intelligence to determine the quality of products by employing a learning model. Then, in the machine learning system 1, products designated as inspection targets are made to flow on the same production line as products for which quality determination is actually performed and products for which the learning model used in the inspection has not yet been generated. Quality determination is performed on the products for which quality determination is performed, and simultaneously, learning data used in generating the learning model is collected for the products for which quality determination is performed.

[0021] The machine learning system 1 of implementation method 1 is described in detail. For example... Figure 1As shown, the machine learning system 1 of Embodiment 1 includes an inspection device 10, an imaging unit (e.g., a camera 20), and a production instruction server 30. Here, the inspection device 10, camera 20, and production instruction server 30 are... Figure 1 The description indicates that these are separate, independent structures, but they can also be installed as a single unit. Furthermore, the production instruction server 30 is mostly a structure that transmits and receives information from the inspection device 10 via communication lines. The camera 20 captures images of the products flowing through the production line. Moreover, this product includes both a quality assessment object and a learning object.

[0022] The inspection device 10 includes a preprocessing unit 11, a setting file storage unit 12, an image storage unit 13, a learning model storage unit 14, an inspection processing unit 15, a display unit 16, and a learning model generation unit 17. Here, the setting file storage unit 12, the image storage unit 13, and the learning model storage unit 14 are, for example, storage devices such as a computer hard drive or an SSD (Solid State Drive). Furthermore, the preprocessing unit 11, the inspection processing unit 15, the display unit 16, and the learning model generation unit 17 can be implemented by a program executed by the computer's computing unit. Additionally, for example, the learning model generation unit 17 can be configured on a cloud server.

[0023] The preprocessing unit 11 generates an inspection object image by cutting out the image of the inspection object part from the product image based on a setting file indicating the position and range of the inspection object part of the product, and saves the generated inspection object image in the image storage unit 13. Here, the setting file is generated in advance by the operator and saved in the setting file storage unit 12. More specifically, the preprocessing unit 11 reads production instruction information from the production instruction server 30 and reads the setting file corresponding to the product information recorded in the production instruction information from the setting file storage unit 12.

[0024] Here, production instruction information and setting documents are explained in detail. Furthermore, the following description will illustrate an example of a product that is inspected, specifically a vehicle or a component of a vehicle. Additionally, both the production instruction information and setting documents are pre-generated by the operator and stored in the production instruction server 30 and the setting document storage unit 12.

[0025] first, Figure 2 This represents an example of production instruction information used in the machine learning system of Implementation 1. Figure 2 The example shown involves the E1 model, which is painted white. For example... Figure 2 As shown, production instruction information is set for each product (e.g., vehicle model). The production instruction server 30, based on the production line flow, appropriately sends the production instruction information for the vehicle model that should be checked at that moment to the preprocessing unit 11. Figure 2 In the example shown, the production instruction information uses vehicle model and paint color as vehicle model information, and the vehicle model information describes the product specifications related to that vehicle model. The product specifications describe the presence or absence of components assembled on the product and the types of those components. In the machine learning system 1 of Embodiment 1, the parts described in the product specifications are used as inspection target parts, and a good or bad judgment is made for each inspection target part. Furthermore, in the machine learning system 1 of Embodiment 1, an inspection instruction identifier is displayed in the production instruction information. This inspection instruction identifier represents either the inspection target or a state that is not yet learned for each inspection target part.

[0026] then, Figure 3 This is an example of the description of the configuration file used in the machine learning system of Implementation 1. Figure 3 The example shown represents the same as Figure 2 This is a portion of the configuration file corresponding to vehicle model E1. For example... Figure 3 As shown, in the setting file, for each inspection object part, the clipping coordinates for cutting out the area containing the inspection object part from the product image obtained from the preprocessing unit 11, the learning model path indicating the location where the learning model is saved, and the image saving path for saving the image of the inspection object part generated by the preprocessing unit 11 through the cutting process are described.

[0027] Here, refer to Figure 4 This explains the preprocessing in the preprocessing unit 11. Therefore, Figure 4 A diagram illustrating the preprocessing of the machine learning system in Implementation Method 1. Figure 4 In the example shown, the preprocessing unit 11 performs cutout processing on the product image acquired by the camera 20 according to the setting file. For example... Figure 4 As shown, the preprocessing unit 11 generates a cut-out image of the portion including the logo and other parts to be inspected, based on the cropping coordinates recorded in the setting file, as an image of the part to be inspected. Then, the preprocessing unit 11 saves the generated image of the part to be inspected in the image save path specified in the setting file.

[0028] Taking the logo as an example, the preprocessing unit 11 from Figure 3The area containing the logo is cut out from the cutting coordinates shown in the setting file, generating an image of the logo's inspection target area. Then, the inspection processing unit 15 performs good / bad judgment processing using artificial intelligence on the image of the inspection target area, where the good / bad judgment object is represented by production instruction information describing the product specifications. More specifically, the inspection processing unit 15 selects a learning model for each inspection target area stored in the image storage unit 13, and uses artificial intelligence based on the selected learning model to determine whether the inspection target area matches the specifications represented by the production instruction information. The display unit 16 displays the inspection results based on the inspection processing unit 15 to the operator, and displays a user interface for the operator to operate the inspection device 10.

[0029] Furthermore, when the preprocessing unit 11 generates images of the areas to be inspected, the inspection processing unit 15 inspects all such images, but skips the inspection of the learning object. This prevents products that are otherwise correct from being judged as defective due to inconsistencies between production instructions and inspection results.

[0030] The learning model generation unit 17 reads the image of the inspection target part, which is used as the learning object in the production instruction information, from the image storage unit 13. Using the read inspection target part image as input, and through machine learning with the production instruction information as supervised data, it generates a learning model capable of identifying the inspection target part image of the learning object. Thus, the artificial intelligence applying the learning model generated by the learning model generation unit 17, when given the inspection target part image of the learning object as input, can output the specifications of the part represented by the production instruction information. Furthermore, the learning model generation unit 17 saves the generated learning model in the learning model saving path specified in the setting file (e.g., the storage area in the learning model storage unit 14).

[0031] Next, the operation of the machine learning system 1 in Implementation 1 will be described. Figure 5 This is a flowchart illustrating the actions of the machine learning system 1 in implementation method 1. Furthermore, in Figure 5 In the example, the machine learning system 1 is shown moving along the same production line as the product to be inspected, representing the actual production quality assessment object and the learning object for which the learning model used in the inspection has not yet been generated. Furthermore, in... Figure 5 The machine learning system 1 shown illustrates a process where a good or bad judgment is made on an object, and simultaneously, learning data is collected for generating a learning model, and the collected learning data is used for learning. However, the learning process using the learning data can also be performed in a different device than the inspection device 10 of the machine learning system 1.

[0032] like Figure 5 As shown, in the machine learning system 1 of Embodiment 1, firstly, the inspection device 10 reads the production instruction information sent from the production instruction server 30 (step S10). Furthermore, in the machine learning system 1, the preprocessing unit 11, the inspection processing unit 15, and the learning model generation unit 17 each read the production instruction information.

[0033] Next, the preprocessing unit 11 reads the setting file saved in the setting file storage unit 12 (step S11). Then, the preprocessing unit 11 uses the camera 20 to capture a product image (step S12), reads the setting file corresponding to the vehicle model recorded in the production instruction information read in step S10, and generates an image of the inspection object part cut out from the captured product image. Furthermore, the preprocessing unit 11 saves the generated image of the inspection object part in the image storage unit 13 (step S13).

[0034] The processing steps S10 to S13 are performed concurrently with the inspection processing using the inspection device 10 and the data collection processing for learning. In other words, in the machine learning system 1, the preprocessing unit 11 is used concurrently for the collection of inspection and learning data.

[0035] Next, in the machine learning system 1 of Embodiment 1, the inspection processing unit 15 reads the image of the inspection target part stored in the image storage unit 13 and performs an inspection. At this time, the inspection processing unit 15 refers to the production instruction information read in step S10 and the inspection instruction flag of the read inspection target part image. If the inspection instruction flag indicates "inspection", inspection processing is performed; if it indicates "not learned", inspection processing is not performed and the next inspection target part image is read. In other words, the inspection processing unit 15 determines whether the read inspection target part image is a component subject to quality judgment. If the judgment is true, inspection processing is performed; if the judgment is false, inspection is not performed (step S14).

[0036] In the inspection process following the "yes" branch in step S14, the inspection processing unit 15 performs AI inspection on the image of the inspection target area using the read learning model, referring to the learning model storage path specified by the setting file (step S15). Furthermore, the inspection processing unit 15 performs inspection processing on all inspection target area images contained in a product image and then outputs the inspection results. Then, in the inspection processing unit 15, if the production instruction information matches the inspection results, it determines that the inspected product is a good product (the "yes" branch in step S16), and saves the image at the location specified by the storage path in the setting file for each model, component, and specification recorded in the production instruction information (step S17). The image saved in step S17 is saved as a production log. Furthermore, in step S16, if the production instruction information and the inspection results are inconsistent (the "no" branch in step S16), after the inspection processing unit 15 issues a warning on the display unit 16 indicating a defective product has been generated (step S18), it saves the image for the production log (step S17).

[0037] On the other hand, in the processing following the "no" branch in step S14, learning processing is performed. During learning processing, the learning model generation unit 17 determines whether it is a learning execution time (step S19). In the machine learning system 1 of Embodiment 1, the learning model generation unit 17 performs the generation and relearning of the learning model at a pre-set period (e.g., a period of six months to one year) or at a time specified by the operator or the system. Then, if the learning model generation unit 17 determines that it is a specified learning time (the "yes" branch in step S19), the learning model generation unit 17 uses production instruction information as monitoring data and images of inspection target parts saved at a location specified by the save path of the setting file as input to generate a learning model (step S20). Furthermore, the learning model generation unit 17 generates a learning model for each combination of vehicle model and inspection target part.

[0038] Then, if it is determined in step S19 that the learning execution timing is not yet complete (the branch in step S19 is no), after the processing in step S20, the processing for the next inspection product is performed again from step S10.

[0039] According to the above description, in the machine learning system 1 of Embodiment 1, the preprocessing unit 11 of the inspection device 10 used in product inspection is used to collect learning data used in the generation of the learning model. There is no need to prepare a separate system for the generation of learning data, which simplifies the system structure.

[0040] Furthermore, in the machine learning system 1 of Embodiment 1, learning data can be collected by having the part of the learning target flow together with the product being inspected on the production line where the product being inspected flows. Therefore, there is no need to set up a separate production line for collecting learning data, thus simplifying the system structure. Moreover, in the machine learning system 1 of Embodiment 1, by collecting learning data by having the product as the learning target flow through the actual production line, image conditions can be made consistent between the actual inspection and learning states, thus enabling the generation of a learning model with high part detection accuracy.

[0041] Furthermore, in the machine learning system 1 of Embodiment 1, since the preprocessing unit 11 can be used to process the image of the inspection object part from the product image by computer, the operator does not need to generate the inspection object part image for learning data by manual operation, which can make the collection of learning data more efficient.

[0042] Furthermore, in the machine learning system 1 of embodiment 1, since the inspection of the object to be inspected can be carried out using only an optical camera as the imaging unit, the complexity of the system structure, such as setting up multiple types of sensors, can be avoided.

[0043] The invention completed by the inventor has been specifically described above according to the embodiments. However, the invention is not limited to the above embodiments, and it is self-evident that various modifications can be made without departing from its spirit.

Claims

1. A machine learning system, having: The photography department is responsible for photographing products and obtaining product images. The preprocessing unit, based on the product image, cuts out the image of the inspection target part according to a setting file indicating the position and range of the inspection target part of the product to generate an inspection target part image, and saves the generated inspection target part image in the image storage unit; The inspection processing unit performs quality / failure judgment processing using artificial intelligence on the image of the inspected part of the object to be judged as good or bad. The object to be judged as good or bad is represented by production instruction information describing the product specifications; and The learning model generation unit takes the learning data as input and the production instruction information related to the learning object as monitoring data to generate a learning model used for judging the quality of the learning object. The preprocessing unit accumulates images of the inspection target parts related to the products of the learning object that are designated as not good or bad objects based on the production instruction information from the inspection target part images stored in the image storage unit, and uses these images as learning data for the learning model applied in the artificial intelligence. The learning model is set for each of the inspected parts. The production instruction information includes an inspection instruction identifier for each of the inspected parts, indicating either the inspected object or a state that is not yet learned. In the inspection and processing unit, The newly added image of the inspection target area in the image storage unit is read in, the learning model corresponding to the inspection target area is read in, and the good or bad judgment process is performed. If the indicator in the production instruction information is found to be an unlearned part, the good or bad judgment process is skipped.

2. The machine learning system according to claim 1, wherein, The preprocessing unit associates the image of the inspected object with the production instruction information and saves it in the image storage unit.

3. The machine learning system according to claim 1 or 2, wherein, The preprocessing unit reads the setting file corresponding to the product from the setting file storage unit according to the production instruction information, and saves the image of the inspection target part in the file path recorded in the setting file.

4. The machine learning system according to claim 1 or 2, wherein, The product is a vehicle or part of a vehicle.

5. The machine learning system according to claim 1 or 2, wherein, The machine learning system is configured to perform a good or bad judgment process on the product in the production line.

6. A learning data collection method, using an inspection device that performs product quality judgment processing in a production line, collects learning data generated by a learning model used in artificial intelligence applications, wherein the learning data collection method performs the following processing: Image processing involves capturing images of products moving along the production line. Preprocessing: For the product image, an image of the inspection object is generated by cutting out the image of the inspection object part according to the setting file representing the position and range of the inspection object part of the product, and the generated image of the inspection object part is saved in the image storage unit; The inspection process involves performing a good or bad judgment process using artificial intelligence on the image of the inspection object part of the object to be judged as good or bad. The object to be judged as good or bad is represented by the production instruction information describing the specifications of the product. as well as The learning model generation process takes the learning data as input and the production instruction information related to the learning object as supervisory data to generate the learning model used in judging the quality of the learning object. In the preprocessing, images of the inspection target parts related to the product of the learning object that are designated as not being judged as good or bad based on the production instruction information, from the inspection target part images stored in the image storage unit, are accumulated in the image storage unit as the learning data for the learning model. The learning model generation process is performed for each of the inspected areas. The production instruction information includes an inspection instruction identifier for each of the inspected parts, indicating either the inspected object or a state of not being learned. In the inspection process, The newly added image of the inspection target area in the image storage unit is read in, the learning model corresponding to the inspection target area is read in, and the good or bad judgment process is performed. If the indicator in the production instruction information is found to be an unlearned part, the good or bad judgment process is skipped.

7. A storage medium storing a learning data collection program, the learning data collection program being executed by a computing unit located within an inspection device and performing the collection of learning data, the inspection device performing product quality judgment processing in a production line, the learning data being used to generate a learning model for application in artificial intelligence using the inspection device, the learning data collection program causing a computer to perform the following processing: Image processing involves capturing images of products moving along the production line. Preprocessing: For the product image, an image of the inspection object is generated by cutting out the image of the inspection object part according to the setting file representing the position and range of the inspection object part of the product, and the generated image of the inspection object part is saved in the image storage unit; The inspection process involves performing a good or bad judgment process using artificial intelligence on the image of the inspection object part of the object to be judged as good or bad. The object to be judged as good or bad is represented by the production instruction information describing the specifications of the product. as well as The learning model generation process takes the learning data as input and the production instruction information related to the learning object as supervisory data to generate the learning model used in judging the quality of the learning object. In the preprocessing, images of the inspection target parts related to the product of the learning object that are designated as not being judged as good or bad based on the production instruction information, from the inspection target part images stored in the image storage unit, are accumulated in the image storage unit as the learning data for the learning model. The learning model generation process is performed for each of the inspected areas. The production instruction information includes an inspection instruction identifier for each of the inspected parts, indicating either the inspected object or a state that is not yet learned. In the inspection process, The newly added image of the inspection target area in the image storage unit is read in, the learning model corresponding to the inspection target area is read in, and the good or bad judgment process is performed. If the indicator in the production instruction information is found to be an unlearned part, the good or bad judgment process is skipped.

Images