Machine learning system, learning data collection method, and learning data collection program

Abstract

To solve such a problem that complication of a system for collecting inspection and learning data cannot be avoided in a conventional system.SOLUTION: A machine learning system according to the present invention comprises: an imaging unit 20 which acquires a product image by imaging a product; a pre-processing unit 11 which generates an inspection object portion image by cutting out an image of an inspection object portion on the basis of a setting file indicating the position and range of the inspection object portion of the product to the product image and stores the generated inspection object portion image in an image storage unit 13; and an inspection processing unit 15 which performs quality determination processing where the artificial intelligence is applied to the inspection object portion image of a quality determination object showing that it is the quality determination object with production instruction information describing specifications of the product. The inspection object portion image about the product designated as a learning object that is not the quality determination object with the production instruction information out of the inspection object portion images stored in the image storage unit 13 is accumulated in the image storage unit 13 as learning data of a learning model applied to the artificial intelligence.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a machine learning system, a learning data collection method, and a learning data collection program, and more particularly to a machine learning system, a learning data collection method, and a machine learning program that generate a learning model used in artificial intelligence that recognizes the shape of an inspection object by using an image of the inspection object. [Background technology]

[0002] In automobile production lines, image-based inspection is sometimes performed in an inspection process to check whether assembled products are assembled according to specifications. In recent years, a technology has been proposed for such inspection that uses artificial intelligence to determine the quality of an object based on an image containing the object. An example of such recognition and inspection technology is disclosed in Patent Document 1.

[0003] The learning device described in Patent Document 1 includes a camera that captures an image of a product sample to acquire image data, a physical property information acquisition unit that acquires physical property information of the sample, and a calculation unit that generates a learning model. The calculation unit is configured to identify a category of the sample based on rule information that associates the physical property information with a category, generate teacher data by associating the identified category with the image data, and generate the learning model by machine learning using the teacher data. The learning model outputs a sample category in response to input of image data of the sample. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 230356 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technology described in Patent Document 1 requires the provision of a sensor other than a camera that simply captures images in order to obtain physical property information, which makes the system more complicated and expensive, and also reduces versatility.

[0006] The present invention has been made to solve such problems, and has as its object to simplify the system configuration. [Means for solving the problem]

[0007] One aspect of the machine learning system of the present invention includes an imaging unit that photographs a product to obtain a product image, a pre-processing unit that cuts out an image of the inspection target part based on a setting file that indicates the position and range of the inspection target part of the product relative to the product image to generate an image of the inspection target part and stores the generated image of the inspection target part in an image storage unit, and an inspection processing unit that applies artificial intelligence to the image of the inspection target part of an object to be subjected to quality judgment that is indicated as a target for quality judgment by production instruction information that describes the specifications of the product, and among the images of the inspection target part stored in the image storage unit, the images of the inspection target part related to products designated as learning objects that are not targets for quality judgment by the production instruction information are accumulated in the image storage unit as learning data for a learning model applied to the artificial intelligence.

[0008] One aspect of the learning data collection method of the present invention is a method for collecting learning data to be used in creating a learning model to be applied to artificial intelligence using an inspection device that performs quality assurance processing on products on a production line, the method comprising the steps of: an imaging process for photographing products flowing down the production line to obtain product images; a pre-processing process for generating images of the inspection target parts by cutting out images of the inspection target parts based on a setting file that indicates the position and range of the inspection target parts of the products from the product images and storing the generated images of the inspection target parts in an image storage unit; and an inspection process for performing quality assurance processing using artificial intelligence on the images of the inspection target parts of objects to be inspected that are indicated as objects to be inspected by production instruction information that describes the specifications of the product; and, among the images of the inspection target parts stored in the image storage unit, images of the inspection target parts related to products that are designated as learning objects that are not objects to be inspected by the production instruction information are accumulated in the image storage unit as learning data for the learning model.

[0009] One aspect of the learning data collection program of the present invention is a learning data collection program that is executed by a calculation unit provided within an inspection device that performs a process of determining whether a product is good or bad on a production line, and that uses the inspection device to collect learning data to be used for creating a learning model to be applied to artificial intelligence. The program performs an imaging process of photographing products flowing down the production line to obtain product images, a pre-processing process of extracting an image of the inspection target part from the product image based on a setting file that indicates the position and range of the inspection target part of the product, to generate an image of the inspection target part and storing the generated image of the inspection target part in an image storage unit, and an inspection process of applying artificial intelligence to the image of the inspection target part of an object to be subjected to quality determination that is indicated as an object to be subjected to quality determination by production instruction information that describes the specifications of the product, and accumulates, among the images of the inspection target part stored in the image storage unit, the images of the inspection target part related to products that are designated as learning objects that are not objects to be subjected to quality determination by the production instruction information in the image storage unit as learning data for the learning model.

[0010] In the machine learning system, the learning data collection method, and the learning data collection program according to the present invention, learning data to be used for generating a learning model is collected on a system that performs an inspection process. Effect of the Invention

[0011] According to the present invention, the machine learning system, the learning data collection method, and the learning data collection program according to the present invention can simplify the system configuration. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram of a machine learning system according to a first embodiment. [Diagram 2] 1 is an example of production instruction information used in the machine learning system according to the first embodiment. [Diagram 3] 1 is an example of the contents of a configuration file used in the machine learning system according to the first embodiment. [Figure 4] FIG. 2 is a diagram for explaining preprocessing of the machine learning system according to the first embodiment. [Diagram 5] 1 is a flowchart illustrating the operation of the machine learning system according to the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In order to clarify the description, the following description and drawings are omitted and simplified as appropriate. In addition, each element shown in the drawings as a functional block performing various processes can be configured in hardware with a CPU (Central Processing Unit), memory, and other circuits, and in software, can be realized by a program loaded into memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof, and are not limited to any one of them. In addition, the same elements are given the same symbols in each drawing, and duplicated explanations are omitted as necessary.

[0014] Moreover, the above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is read into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray (registered trademark) disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals.

[0015] First embodiment First, a schematic diagram of a machine learning system 1 according to the first embodiment is shown in Fig. 1. As shown in Fig. 1, the machine learning system 1 according to the first embodiment includes an inspection device 10 that judges the quality of a product by artificial intelligence using a learning model. In the machine learning system 1, as products to be inspected, objects to be actually produced for which quality judgment is to be performed and learning objects for which a learning model to be used in the inspection has not yet been created are sent on the same production line, and the machine learning system 1 judges the quality of the objects to be judged for quality while collecting learning data to be used for generating a learning model for the learning objects.

[0016] A machine learning system 1 according to the first embodiment will be described in detail. As shown in FIG. 1, the machine learning system 1 according to the first embodiment includes an inspection device 10, an imaging unit (e.g., a camera 20), and a production instruction server 30. Here, the inspection device 10, the camera 20, and the production instruction server 30 are illustrated in FIG. 1 as independent components located at separate locations, but these devices may be implemented as a single device. Note that the production instruction server 30 is often configured to transmit and receive information to and from the inspection device 10 via a communication line. The camera 20 captures images of products flowing through a production line to obtain product images. These products include objects to be judged pass / fail and objects to be learned.

[0017] The inspection device 10 has a preprocessing unit 11, a setting file storage unit 12, an image storage unit 13, a learning model storage unit 14, an inspection processing unit 15, a display unit 16, and a learning model generation unit 17. Here, the setting file storage unit 12, the image storage unit 13, and the learning model storage unit 14 are, for example, storage devices such as a computer hard disk or SSD (Solid State Drive). In addition, the preprocessing unit 11, the inspection processing unit 15, the display unit 16, and the learning model generation unit 17 can be realized by a program executed by a computing unit of a computer. In addition, the learning model generation unit 17 may be arranged on, for example, a cloud server.

[0018] The preprocessing unit 11 generates an image of the inspection target part by cutting out an image of the inspection target part based on a setting file that indicates the position and range of the inspection target part of the product with respect to the product image, and stores the generated image of the inspection target part in the image storage unit 13. Here, the setting file is created in advance by an operator and stored in the setting file storage unit 12. More specifically, the preprocessing unit 11 reads production instruction information from the production instruction server 30, and reads from the setting file storage unit 12 a setting file that corresponds to the product information described in the production instruction information.

[0019] Here, the production instruction information and the setting file will be described in detail. In the following description, an example will be described in which a vehicle or a component part of a vehicle is the product to be inspected. The production instruction information and the setting file are both created in advance by an operator and stored in the production instruction server 30 and the setting file storage unit 12.

[0020] First, FIG. 2 shows an example of production instruction information used in the machine learning system according to the first embodiment. The example shown in FIG. 2 is related to a vehicle model E1 painted in white. As shown in FIG. 2, production instruction information is provided for each product (for example, vehicle model), and the production instruction server 30 transmits production instruction information for the vehicle model to be inspected at that time to the preprocessing unit 11 as appropriate according to the flow of the production line. In the example shown in FIG. 2, the production instruction information describes the vehicle model and paint color as vehicle model information, and the product specifications related to the vehicle model are described as vehicle model information. The product specifications describe the presence or absence of parts assembled in the product and the types of parts. In the machine learning system 1 according to the first embodiment, the parts described in the product specifications are the parts to be inspected, and a pass / fail judgment is performed for each of the parts to be inspected. In addition, in the machine learning system 1 according to the first embodiment, an inspection instruction flag is indicated in the production instruction information. This inspection instruction flag indicates whether each of the parts to be inspected is in an inspection target state or an unlearned state.

[0021] Next, Fig. 3 shows an example of the contents of a setting file used in the machine learning system according to the first embodiment. The example shown in Fig. 3 shows a part of a setting file corresponding to the vehicle model E1 shown in Fig. 2. As shown in Fig. 3, in the setting file, for each inspection target part, there are described cropping coordinates for cutting out an area including the inspection target part from the product image acquired by the pre-processing unit 11, a learning model path indicating the location where the learning model is saved, and an image saving path for saving the inspection target part image generated by the pre-processing unit 11 through the cutting out process.

[0022] Here, the preprocessing in the preprocessing unit 11 will be described with reference to FIG. 4. FIG. 4 is a diagram for explaining the preprocessing in the machine learning system according to the first embodiment. In the example shown in FIG. 4, the preprocessing unit 11 performs a cutout process on a product image acquired by the camera 20 based on a setting file. As shown in FIG. 4, the preprocessing unit 11 generates a cutout image of a portion including an inspection target part such as an emblem based on the trimming coordinates described in the setting file as an inspection target part image. Then, the preprocessing unit 11 saves the generated inspection target part image in the image save path described in the setting file.

[0023] Taking an emblem as an example, the preprocessing unit 11 cuts out the range of the area including the emblem from the trimming coordinates indicated in the setting file in Fig. 3 to generate an image of the emblem's inspection target portion. Then, the inspection processing unit 15 performs a quality judgment process using artificial intelligence on the image of the inspection target portion of the object to be judged, which is indicated to be a quality judgment target by the production instruction information describing the product specifications. More specifically, the inspection processing unit 15 selects a learning model for each inspection target portion stored in the image storage unit 13, and judges whether the inspection target portion matches the specifications indicated in the production instruction information by the artificial intelligence using the selected learning model. The display unit 16 displays the inspection results by the inspection processing unit 15 to the operator, and displays a user interface for the operator to operate the inspection device 10.

[0024] When the preprocessing unit 11 generates images of the parts to be inspected, the inspection processing unit 15 inspects all the images of the parts to be inspected, but skips the inspection of the learning object. This prevents a genuine product from being judged as defective due to an erroneous mismatch between the production instruction information and the inspection result.

[0025] The learning model generation unit 17 reads out the inspection target part image designated as the learning target in the production instruction information from the image storage unit 13, and generates a learning model capable of discriminating the inspection target part image of the learning target by machine learning using the read inspection target part image as input and the production instruction information as teacher data. This enables the artificial intelligence to which the learning model generated by the learning model generation unit 17 is applied to output the specifications of the part indicated in the production instruction information when the inspection target part image of the learning target is input. In addition, the learning model generation unit 17 stores the generated learning model in a learning model storage path described in the setting file (for example, a storage area in the learning model storage unit 14).

[0026] Next, the operation of the machine learning system 1 according to the first embodiment will be described. FIG. 5 shows a flowchart for explaining the operation of the machine learning system 1 according to the first embodiment. FIG. 5 shows the operation of the machine learning system 1 in the case where an object to be actually produced for which a quality judgment is to be made and a learning object for which a learning model to be used for the inspection has not yet been created are flown on the same production line as products to be inspected. In addition, the operation of the machine learning system 1 shown in FIG. 5 shows a case where the object to be judged for quality judgment is judged for quality judgment, while learning data to be used for generating a learning model for the learning object is collected, and learning is performed using the collected learning data. However, the learning process using the learning data may be performed by a device other than the inspection device 10 of the machine learning system 1.

[0027] 5, in the machine learning system 1 according to the first embodiment, first, the inspection device 10 reads the production instruction information transmitted from the production instruction server 30 (step S10). In the machine learning system 1, the preprocessing unit 11, the inspection processing unit 15, and the learning model generation unit 17 each read the production instruction information.

[0028] Next, the pre-processing unit 11 reads the setting file stored in the setting file storage unit 12 (step S11). After that, the pre-processing unit 11 takes a product image using the camera 20 (step S12), reads the setting file corresponding to the vehicle model described in the production instruction information read in step S10, and generates an inspection target part image by cutting out the inspection target part from the photographed product image. In addition, the pre-processing unit 11 stores the generated inspection target part image in the image storage unit 13 (step S13).

[0029] The processes in steps S10 to S13 are performed in common in the inspection process and the learning data collection process using the inspection device 10. That is, in the machine learning system 1, the preprocessing unit 11 is commonly used for the inspection and the collection of learning data.

[0030] Next, in the machine learning system 1 according to the first embodiment, the inspection processing unit 15 reads out the inspection target part image stored in the image storage unit 13 and performs the inspection. At this time, the inspection processing unit 15 refers to the production instruction information read in at step S10, refers to the inspection instruction flag of the read inspection target part image, and performs the inspection process if the inspection instruction flag is set to inspection, and reads the next inspection target detection part image without performing the inspection process if it is set to unlearned. In other words, the inspection processing unit 15 judges whether the read inspection target part image is a part to be judged pass / fail, and performs the inspection process if the judgment is true, and does not perform the inspection if the judgment is false (step S14).

[0031] In the inspection process following the YES branch of step S14, the inspection processing unit 15 refers to the learning model storage path specified in the setting file and performs AI inspection on the inspection target part image using the read learning model (step S15). The inspection processing unit 15 performs inspection processing on all the inspection target part images included in one product image and then outputs the inspection result. Then, in the inspection processing unit 15, if the production instruction information and the inspection result match, the inspected product is determined to be a non-defective product (YES branch of step S16), and the image is stored in a location specified by the storage path of the setting file for each vehicle model, part, and specification described in the production instruction information (step S17). The image stored in this step S17 is stored as a production log. Also, in step S16, if the production instruction information and the inspection result do not match (NO branch of step S16), the inspection processing unit 15 issues a warning on the display unit 16 that a defective product has occurred (step S18), and then stores the image for the production log (step S17).

[0032] On the other hand, in the process following the NO branch of step S14, a learning process is performed. In the learning process, the learning model generation unit 17 judges whether or not it is time to execute learning (step S19). In the machine learning system 1 according to the first embodiment, the learning model generation unit 17 executes generation of a learning model and re-learning of the learning model at a preset cycle (for example, a cycle of six months to one year) or at a timing designated by an operator or the system. Then, when the learning model generation unit 17 judges that it is the predetermined learning timing (YES branch of step S19), the learning model generation unit 17 uses the product instruction information as teacher data and inputs the image of the inspection target part saved in the location designated by the save path of the setting file to generate a learning model (step S20). Note that the learning model generation unit 17 generates a learning model for each combination of the vehicle type and the inspection target part.

[0033] After the process of step S17, if it is determined in step S19 that it is not yet time to execute learning (NO in step S19), the process for the next inspected product is repeated from step S10 after the process of step S20.

[0034] As explained above, in the machine learning system 1 of embodiment 1, the pre-processing unit 11 of the inspection device 10 used for product inspection is used to collect learning data to be used in generating a learning model, thereby eliminating the need to prepare a separate system for generating learning data and simplifying the system configuration.

[0035] Furthermore, in the machine learning system 1 according to the first embodiment, since learning data can be collected by flowing the parts to be learned together with the products to be inspected on the production line on which the products to be inspected flow, there is no need to provide a separate line for collecting learning data, and the system configuration can be simplified. Furthermore, since the machine learning system 1 according to the first embodiment collects learning data by flowing the products to be learned on the production line on which actual production is performed, image acquisition conditions can be matched between the actual inspection and the learning state, and therefore a learning model with high part detection accuracy can be generated.

[0036] Furthermore, in the machine learning system 1 according to the first embodiment, the extraction of images of the inspection area from the product image can be performed by computer processing using the pre-processing unit 11, eliminating the need for an operator to manually generate images of the inspection area for learning data, thereby making it possible to efficiently collect learning data.

[0037] Furthermore, in the machine learning system 1 according to the first embodiment, inspection of the inspection target area can be performed using only an optical camera as the imaging unit, thereby preventing the system configuration from becoming complicated, such as by installing multiple types of sensors.

[0038] The invention made by the inventor has been specifically described above based on an embodiment, but it goes without saying that the present invention is not limited to the embodiment already described, and various modifications are possible without departing from the gist of the invention. [Explanation of symbols]

[0039] 1. Machine Learning Systems 10 Inspection equipment 11 Pretreatment section 12 Configuration file storage section 13 Image storage section 14 Learning model storage unit 15 Inspection processing section 16 Display 17 Learning model generation unit 20 Camera 30 Production instruction server

Claims

1. an imaging unit that photographs the product to obtain a product image; a pre-processing unit that cuts out an image of the inspection target portion of the product from the product image based on a setting file that indicates the position and range of the inspection target portion of the product, generates an inspection target portion image, and stores the generated inspection target portion image in an image storage unit; an inspection processing unit that performs a quality determination process by applying artificial intelligence to the image of the inspection target portion of the object to be subjected to quality determination, which is indicated as a quality determination target by production instruction information describing the specifications of the product; a learning model generation unit that receives learning data as an input, and generates a learning model to be used for determining whether the learning object is good or bad using the production instruction information related to the learning object as teacher data, the preprocessing unit accumulates, in the image storage unit, the images of the inspection target portion related to the product designated as a learning object that is not a pass / fail judgment object by the production instruction information, among the images of the inspection target portion stored in the image storage unit, as the learning data of the learning model applied to the artificial intelligence; The learning model is provided for each of the inspection target parts, The production instruction information includes an inspection instruction flag indicating whether each of the inspection target portions is in an inspection target state or an unlearned state, The inspection processing unit includes: The image of the inspection target portion newly added to the image storage unit is read, the learning model corresponding to the inspection target portion is read, and the pass / fail judgment process is performed. A machine learning system that skips the pass / fail judgment process for parts in the production instruction information for which the inspection instruction flag has not been learned.

2. The machine learning system according to claim 1 , wherein the preprocessing unit stores the image of the inspection target portion in the image storage unit in association with the production instruction information.

3. The machine learning system of claim 1 or 2, wherein the preprocessing unit reads a setting file corresponding to the product from a setting file storage unit based on the production instruction information, and saves the image of the inspection target area in a file path described in the setting file.

4. The machine learning system of claim 1 , wherein the product is a vehicle or a part of a vehicle.

5. The machine learning system according to claim 1 , wherein the machine learning system is configured to perform a quality determination process for the products on a production line.

6. A method for collecting learning data to be used for creating a learning model to be applied to artificial intelligence using an inspection device that performs quality control processing on a production line, comprising: An imaging process for capturing images of products flowing through a production line; a pre-processing step of extracting an image of the inspection target portion from the product image based on a setting file indicating the position and range of the inspection target portion of the product, thereby generating an inspection target portion image, and storing the generated inspection target portion image in an image storage unit; an inspection process for performing a quality judgment process by applying artificial intelligence to an image of an inspection target portion of an object to be judged as being a quality judgment target, the image being indicated as being a quality judgment target by production instruction information describing the specifications of the product; a learning model generation process for generating the learning model to be used for determining whether the learning object is good or bad using the learning data as input and the production instruction information related to the learning object as teacher data; In the pre-processing, among the images of the inspection target portion stored in the image storage unit, images of the inspection target portion related to a product designated as a learning object that is not a pass / fail judgment object by the production instruction information are accumulated in the image storage unit as the learning data of the learning model; The learning model generation process is performed for each of the inspection target portions, The production instruction information includes an inspection instruction flag indicating whether each of the inspection target portions is in an inspection target state or an unlearned state, In the inspection process, The image of the inspection target portion newly added to the image storage unit is read, the learning model corresponding to the inspection target portion is read, and the pass / fail judgment process is performed. The learning data collection method skips the pass / fail determination process for a portion for which an inspection instruction flag in the production instruction information is unlearned.

7. A learning data collection program that is executed by a calculation unit provided in an inspection device that performs a quality determination process for products on a production line, and that collects learning data to be used in creating a learning model to be applied to artificial intelligence using the inspection device, An imaging process for capturing images of products flowing through a production line; a pre-processing step of extracting an image of the inspection target portion from the product image based on a setting file indicating the position and range of the inspection target portion of the product, thereby generating an inspection target portion image, and storing the generated inspection target portion image in an image storage unit; an inspection process for performing a quality judgment process by applying artificial intelligence to an image of an inspection target portion of an object to be judged as being a quality judgment target, the image being indicated as being a quality judgment target by production instruction information describing the specifications of the product; a learning model generation process for generating the learning model to be used for determining whether the learning object is good or bad using the learning data as input and the production instruction information related to the learning object as teacher data; In the pre-processing, among the images of the inspection target portion stored in the image storage unit, images of the inspection target portion related to a product designated as a learning object that is not a pass / fail judgment object by the production instruction information are accumulated in the image storage unit as the learning data of the learning model; The learning model generation process is performed for each of the inspection target portions, The production instruction information includes an inspection instruction flag indicating whether each of the inspection target portions is in an inspection target state or an unlearned state, In the inspection process, The image of the inspection target portion newly added to the image storage unit is read, the learning model corresponding to the inspection target portion is read, and the pass / fail judgment process is performed. The learning data collection program skips the pass / fail determination process for a portion for which an inspection instruction flag in the production instruction information is unlearned.

Images