Teacher data generation device, teacher data generation method, and computer program

Abstract

To achieve manpower saving, accuracy increase and processing amount improvement in processing from machine learning to automatic selection of an object by achieving automation in machine learning.SOLUTION: A teacher data generation device comprises: an imaging unit which acquires an image of an object being a selection object; a position acquisition unit which acquires position information of the object by using the image acquired by the imaging unit; a material acquisition unit which acquires material information of the object by measuring the material of the object using the position information of the object; and a teacher data generation unit which generates teacher data in which the image of the object, the position information and the material information are associated with each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a teacher data generating device, a teacher data generating method, and a computer program. [Background technology]

[0002] In order to recycle metals such as aluminum alloys from scrap, a technique for separating these metals is known (see, for example, Patent Document 1). The technique described in Patent Document 1 irradiates the scrap with a laser beam and separates valuable materials using reflectance data of the laser beam.

[0003] Meanwhile, a technique for detecting the position and orientation of an object by machine learning is known. Furthermore, a learning data generation device that generates training data used for performing supervised machine learning is known (see, for example, Patent Document 2). The device described in Patent Document 2 creates computer graphics (CG) of an object appearing in an acquired background image. This device places the created CG on the background image and performs rendering to create a composite rendered image. This device generates training data that links the rendered image with the position and orientation of the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6635423 [Patent Document 2] Patent No. 6675691 Summary of the Invention [Problem to be solved by the invention]

[0005] Conventionally, when scrap pieces flowing on a line are visually selected by an operator, it is necessary for the operator to undergo several months of training in order to select scrap pieces having similar appearances, which may result in high labor costs. In addition, even trained operators may make selection errors. In order to solve such problems, there is a need for labor saving in a system for selecting specific metals and the like from scrap pieces (for example, Patent Document 1). However, in the technology described in Patent Document 1, it is necessary to irradiate all scrap pieces to be selected with a laser beam, which requires a long time for the selection process. In addition, since the selection is performed only based on the information on the emission spectrum of the laser beam, there is a risk of a selection error occurring if the emission spectrum cannot be properly acquired due to an error in the focus adjustment of the laser beam or due to dirt on the target object.

[0006] In addition, in order to reduce the labor required for the sorting system, automatic sorting may be performed using a learning model created by machine learning as described in Patent Document 2. In automatic sorting, since a machine automatically performs sorting, labor saving in sorting processing and improved sorting accuracy can be expected. On the other hand, if the material of each object is analyzed to determine whether it is recyclable or not, the analysis may take time and the improvement of the sorting processing volume may not be expected. In other words, although labor saving and high accuracy can be expected by introducing automatic sorting technology, the improvement of the sorting processing volume becomes an issue. Therefore, in order to improve the sorting processing volume while achieving both labor saving and high accuracy, it is preferable to prepare a huge amount of training data during machine learning. However, when training data for creating a learning model is generated manually, even if labor saving can be achieved by automatic sorting, a lot of manpower is required during the machine learning stage before that.

[0007] The present invention has been made to solve the above-mentioned problems, and aims to achieve automation during machine learning, thereby reducing the labor required, improving accuracy, and increasing the throughput in the processes from machine learning to automatic sorting of objects. [Means for solving the problem]

[0008] The present invention has been made to solve at least one of the above problems, and can be realized in the following forms. A teacher data generation device comprising: an imaging unit that captures an image of an object to be selected, a position acquisition unit that acquires position information of the object using the image acquired by the imaging unit, a material acquisition unit that acquires material information of the object by a laser induced breakdown spectrometer that measures the material of the object using the position information of the object, a teacher data generation unit that generates teacher data in which the image of the object, the position information, and the material information are associated with each other, and a teacher data extension unit that generates one or more extended teacher data using the teacher data generated by the teacher data generation unit, wherein the teacher data extension unit performs image processing on the image of the object included in the original teacher data, and generates the extended teacher data by associating the image of the object after the image processing with the position information and the material information included in the original teacher data. The present invention can also be realized in the following forms.

[0009] (1) According to one aspect of the present invention, there is provided a teacher data generation device comprising: an imaging unit that captures an image of an object to be selected, a position acquisition unit that acquires position information of the object using the image acquired by the imaging unit, a material acquisition unit that acquires material information of the object by measuring a material of the object using the position information of the object, and a teacher data generation unit that generates teacher data that associates the image of the object, the position information, and the material information.

[0010] According to this configuration, the teacher data generating unit generates teacher data that associates the image, position information, and material information of the object, which are automatically acquired by the photographing unit, the position acquiring unit, and the material acquiring unit. Since a huge amount of teacher data is required to create a learning model, the time required to create a learning model (learning process) can be reduced by automating the generation of teacher data as in this configuration. Furthermore, automatic sorting using a learning model can achieve more accurate sorting than sorting by an operator, and can speed up sorting. In other words, according to the teacher data generating device of this configuration, it is possible to reduce the number of people, improve accuracy, and increase the amount of processing in the processes from machine learning to automatic sorting of objects.

[0011] (2) In the teacher data generation device of the above aspect, the device may further include a teacher data extension unit that generates one or more extended teacher data using the teacher data generated by the teacher data generation unit, and the teacher data extension unit may perform image processing on an image of the object included in the original teacher data, and generate the extended teacher data by matching the image of the object after the image processing with the position information and the material information included in the original teacher data. According to this configuration, the teacher data extension unit generates extended teacher data by performing image processing on the original teacher data, so that extended teacher data that has been modified can be easily generated, and a large amount of extended teacher data can be generated without hassle. As a result, a learning model is created using more extended teacher data than the teacher data, so that the detection rate of objects in automatic selection using the learning model can be improved.

[0012] (3) In the teacher data generation device of the above aspect, the teacher data extension unit may perform the image processing by selecting multiple original teacher data, generating cropped images by cutting out a predetermined area including the object for each image of the object included in the selected teacher data, rotating the cropped images at a random angle, and generating a composite image by pasting the cropped images at a randomly determined position on a single background image without overlapping. According to this configuration, the extended teacher data generated by the teacher data extension unit contains multiple original teacher data, and therefore the number of teacher data is greater than that of the teacher data. Therefore, the learning model created using the extended teacher data of this configuration can achieve an improved processing volume compared to the learning model created using teacher data when multiple selection objects are automatically selected in a state where they are transported separately.

[0013] (4) In the teacher data generation device of the above aspect, the teacher data extension unit may generate the extended teacher data so that the number of objects associated with all of the generated extended teacher data is approximately the same for each type of material of the objects. With this configuration, even if the number of object data having specific material information as training data is small, the bias of the material information contained in the original training data included in all of the extended training data is reduced. As a result, the detection rate of the selection target can be improved in automatic selection of objects using a learning model created using the extended training data.

[0014] (5) The teacher data generation device according to the above aspect may further include a transport unit that transports the object. According to this configuration, the position information and material information of the object are automatically acquired while the positions of the photographing unit and the position acquisition unit are fixed.

[0015] (6) In the teacher data generation device of the above aspect, the transporting unit may transport multiple objects to be discriminated in a row along the transporting direction so that one object is included in one image acquired by the photographing unit. With this configuration, the teacher data generation unit can easily generate teacher data that associates images of objects.

[0016] (7) In the teacher data generation device of the above aspect, the teacher data generation unit may generate the teacher data that corresponds, for one of the objects, an image of the object, the position information, and the material information. According to this configuration, images containing multiple objects are not used to generate the training data, making it easier to perform the matching.

[0017] (8) In the teacher data generation device of the above aspect, the material acquisition unit may be a laser induced breakdown spectrometer. According to this configuration, the time required to identify the material of the object can be reduced.

[0018] (9) In the teacher data generation device of the above aspect, the position acquisition unit may acquire, as the position information, a conveying speed of the conveying unit and height information of the object conveyed by the conveying unit, and the material acquisition unit may automatically adjust the focus of the laser-induced breakdown spectrometer using the position information. According to this configuration, the acquisition of position information and material information of the object transported by the transport unit is automated, thereby reducing the number of people required when generating teacher data.

[0019] (10) In the teacher data generation device of the above aspect, the object may be metal scrap waste. Since the shape of waste material retains the characteristics of the original shape, and the specific parts that are the source of waste material are made of specific materials, there is often a correlation between the shape and material of the waste material. In this configuration, by recognizing in advance what kind of waste the target object is, training data is created that associates an image showing the shape of the target object with material information of the target object. As a result, by using the training data generation device of this configuration, it is possible to improve the amount of sorting processing when a learning model using training data automatically sorts the material information of an object based on an image of the object.

[0020] (11) In the teacher data generation device of the above aspect, the photographing unit may photograph the scrap at a resolution that captures the scrap in an area of ​​10 pixels by 10 pixels or more. Since the imaging unit with this configuration can capture images of objects larger than a predetermined size, the training data generation unit creates training data for creating a learning model with higher accuracy.

[0021] (12) In the teacher data generation device of the above aspect, the position acquisition unit may identify the position of the object appearing in the image by performing image recognition on the image acquired by the imaging unit. According to this configuration, by identifying the position of the object in the image, the training data generation unit can generate training data for creating a learning model that performs highly accurate selection.

[0022] (13) In the teacher data generation device of the above aspect, when the material information acquired by the material acquisition unit satisfies a predetermined material standard, the teacher data generation unit generates the teacher data using the object that satisfies the material standard, and when the material information does not satisfy the material standard, it is not necessary to generate the teacher data associated with the object that does not satisfy the material standard. According to this configuration, since the generation of teacher data is determined based on a certain material criterion, teacher data associated with an unclear material as material information is not generated. In other words, the data generator 24 can create highly accurate teacher data.

[0023] The present invention can be realized in various forms, for example, in the form of a teacher data generation device, a learning device, a learning model generation device, a selection device, and a system equipped with these devices, a teacher data generation system, and a teacher data generation method, a computer program for executing these devices and methods, a server device for distributing this computer program, and a non-transitory storage medium on which a computer program is stored. [Brief description of the drawings]

[0024] [Figure 1] 1 is a schematic diagram of a learning model creation system according to an embodiment of the present invention. [Diagram 2] FIG. 4 is an explanatory diagram of teacher data generated by a data generation unit. [Diagram 3] FIG. 11 is an explanatory diagram of extended data generated by a data extension unit. [Figure 4] FIG. 1 is an explanatory diagram of one piece of extended data including a composite image. [Diagram 5] 1 is a flowchart of an extended teacher data generation method. [Figure 6] 13 is a table showing the relationship between the number of training data and extended training data and the non-detection rate of scrap material. [Figure 7] 13 is a table showing the relationship between the ratio of materials contained in the training data and the extended training data and the non-detection rate. [Figure 8] 13 is a table showing the generation times of teacher data in an embodiment and a comparative example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] <Embodiment> 1. Configuration of the learning model creation system: FIG. 1 is a schematic diagram of a learning model creation system 100 according to an embodiment of the present invention. The learning model creation system (teacher data generation device) 100 generates teacher data necessary for creating a learning model, and creates a learning model to be used for automatic selection using the generated teacher data. The learning model creation system 100 of this embodiment generates data in which an image of an object, material information of the object, and position information of the object are associated with each other as teacher data to be used when creating a learning model. As a result, the learning model creation system 100 realizes labor saving and the like in the processes from the creation of teacher data to the automatic selection of objects using a learning model.

[0026] As shown in FIG. 1, the learning model creation system 100 includes a belt conveyor (conveying section) 10 that transports multiple objects OB to be selected, an imaging section 30 that acquires images of each of the transported objects OB, a material acquisition section 40 that irradiates each of the objects OB transported by the belt conveyor 10 with a laser LS to acquire material information of the objects OB, and a model creation device 20 that creates teacher data and a learning model using the images of the objects OB and the material information of the objects OB.

[0027] The belt conveyor 10 includes a belt 11 that conveys objects OB in the conveying direction indicated by the arrow in Fig. 1. The belt conveyor 10 of this embodiment conveys a plurality of objects OB in a line on the belt 11 along the conveying direction so that one object OB is included in one image acquired by the photographing unit 30. In this embodiment, the plurality of objects OB are conveyed in a line at intervals of 30 cm along the conveying direction.

[0028] The photographing unit 30 is a stereo camera that photographs each object OB transported on the belt 11. The photographing unit 30 photographs each object OB with a resolution that captures the object OB in an area of ​​10 pixels x 10 pixels or more. The photographing unit 30 photographs the belt 11 at regular photographing intervals.

[0029] The material acquisition unit 40 is a laser induced breakdown spectrometer. The material acquisition unit 40 automatically adjusts the focal distance to the object OB using height information included in the position information to the object OB on the belt 11 acquired by the position acquisition unit 25 described later. In addition, the material acquisition unit 40 adjusts the irradiation timing of the laser LS using the conveying speed of the belt conveyor 10 acquired by the position acquisition unit 25.

[0030] The material acquisition unit 40 collects and separates the radiation light of the laser-induced plasma generated from the surface of the object OB by irradiating the laser LS to each object OB. The material acquisition unit 40 calculates the elements contained in the object OB by comparing the obtained emission spectral line intensity with the spectral line intensity of a specific component element. The object OB in this embodiment is a metal scrap as waste. Since the object OB is a scrap dismantled from a specific part, it is known in advance that the material of each object OB is one of a limited number of types of material. Therefore, by selecting a specific component element of interest in advance, the material acquisition unit 40 can acquire material information of the object OB by laser-induced breakdown spectroscopy. In the following, the material information of the object OB is also simply referred to as the "material of the object OB".

[0031] In this embodiment, the selection of a plurality of objects OB including cast materials and wrought materials that can be distinguished by the concentration of Si (silicon) will be described. The material acquisition unit 40 of this embodiment selects materials based on the ratio of the spectral line intensity at a wavelength of 671 nm (nanometers) corresponding to Si to the spectral line intensity at a wavelength of 309 nm corresponding to Al (aluminum). The material acquisition unit 40 identifies the material of the object OB in which the spectral line intensity of Si is greater as cast materials, and identifies the material of the object OB in which the spectral line intensity of Al is greater as wrought materials. Note that a 1064 nm Nd:YAG pulse laser (pulse width 6 ns) is used as the laser oscillator, and the laser light energy is 35.6 mJ. The spectrometer targets wavelengths of 200 to 740 nm.

[0032] The model creating device 20 is connected by wires to an imaging unit 30 and a material acquisition unit 40. The model creating device 20 includes a personal computer (PC) 21 and a monitor 28 that displays various images in response to control signals from the PC 21. The PC 21 includes a storage unit 27 that stores various data, and a control unit 22 that performs various controls such as saving data in the storage unit 27.

[0033] The storage unit 27 is composed of a hard disk drive (HDD: Hard Disk Drive) etc. The storage unit 27 includes a teacher data database 271 (teacher data DB) 271 that stores teacher data generated by the data generation unit 24, an extended data database (extended data DB) 272 that stores extended teacher data generated by the data extension unit 23, and a model database (model DB) 273 that stores a learning model created by the learning unit 26 using the extended teacher data. The data generation unit 24, the data extension unit 23, and the learning unit 26 will be described later.

[0034] The control unit 22 functions by a central processing unit (CPU) (not shown) expanding a program stored in a read only memory (ROM) into a random access memory (RAM). The control unit 22 functions as a position acquisition unit 25 that acquires position information of the object OB using an image of each object OB acquired by the imaging unit 30, a data generation unit (teacher data generation unit) 24 that generates teacher data, a data extension unit (teacher data extension unit) 23 that generates extended teacher data using the teacher data generated by the data generation unit 24, and a learning unit 26 that creates a learning model using the extended teacher data generated by the data extension unit 23.

[0035] The position acquisition unit 25 acquires position information of each object OB conveyed by the belt conveyor 10 by performing image recognition on the image of each object OB acquired by the photographing unit 30, which is a stereo camera. The position acquisition unit 25 acquires position information of the object OB in the image by performing image processing (grayscaling, blurring, binarization, contour extraction) on the image in which the object OB appears, and extracting a rectangular area (also called a "bounding box") surrounding the object OB. The position information includes position coordinates of multiple feature points of the object OB in the image. The position coordinates of the feature points are coordinate values ​​of feature points such as edges that represent the shape of the object OB, based on the coordinate axes set in the image. The position acquisition unit 25 can acquire height information of the object OB conveyed on the belt 11 from the position information of the object OB. The height information of the object OB is used by the material acquisition unit 40 to calculate the focal distance to the object OB. The position acquisition unit 25 also acquires the conveying speed of the belt conveyor 10 by using a rotary encoder as a speed sensor.

[0036] The data generating unit 24 generates teacher data that associates the image of each object OB acquired by the imaging unit 30 with the position information acquired by the position acquiring unit 25 and the material information acquired by the material acquiring unit 40. FIG. 2 is an explanatory diagram of the teacher data generated by the data generating unit 24. FIG. 2 shows a part of the teacher data associated with each sample whose object ID is identified as samples SM1 to SM4 among the multiple objects OB. The images of the samples SM1 to SM4 as the objects OB are images IM1 to IM4 shown in FIGS. 1 and 2.

[0037] 2, the teacher data includes an image and position information of one object OB acquired by the imaging unit 30, and material information acquired by the material acquisition unit 40. For example, as the position information of the sample SM1, (X, Y, Z) = (xx1, yy1, zz1) is stored as the coordinate values ​​of the characteristic point P1A. In the material information shown in FIG. 1, either a cast material or a wrought material is associated as the material of each of the samples SM1 to SM4 identified by the material acquisition unit 40. The data generation unit 24 stores the generated teacher data in the teacher data DB 271 of the storage unit 27.

[0038] The data extension unit 23 generates extended teacher data (hereinafter, simply referred to as "extended data") in which a plurality of teacher data are associated as one piece of data, using the teacher data generated by the data generation unit 24. In this embodiment, the data extension unit 23 performs image processing (trimming and rotation) on an image of the object OB included in the original teacher data, as the extended data, and generates data in which the image of the object OB after the image processing is associated with position information and material information included in the original teacher data.

[0039] 3 is an explanatory diagram of the extended data generated by the data extension unit 23. In FIG. 3, a composite image IM having four objects included in the extended data is shown. ED is shown. The data extension unit 23 of this embodiment randomly selects four teacher data from the teacher data DB 271 as a plurality of original teacher data. The data extension unit 23 selects two teacher data of cast material and two teacher data of wrought material to be associated with the extended data so that the number of objects OB whose material is cast material is approximately equal to the number of objects OB whose material is wrought material. In the following, an example will be described in which the four teacher data whose object IDs are SM1 to SM4 shown in FIG. 2 are selected as data to be included in the extended data.

[0040] The data extension unit 23 generates trimmed images by cutting out a predetermined area including each of the images IM1 to IM4 of the object OB included in the selected teacher data. In this embodiment, the data extension unit 23 divides one rectangular background image IMB prepared in advance into four areas AR1 to AR4 (areas separated by dashed lines in FIG. 3), and generates trimmed images that are smaller than the areas AR1 to AR4. The data extension unit 23 pastes the trimmed images of each of the images IM1 to IM4 into areas randomly determined for the four divided areas AR1 to AR4 without overlapping. At this time, the data extension unit 23 pastes trimmed images of images IM1a to IM4a obtained by randomly rotating the images IM1 to IM4 as teacher data into the areas AR1 to AR4, to generate a composite image IM ED Generate.

[0041] The data extension unit 23 of this embodiment generates a composite image IM shown in FIG. ED and the composite image IM ED The position information and material information of the four objects OB in the composite image IM are associated with those in the original training data. ED 4 is an explanatory diagram of one extended data ED1 including the above. In the extended data ED1 shown in FIG. 4, material information included in samples SM1 to SM4, which are the original teacher data, is associated with position information calculated from position information included in the original teacher data by using trimming and rotation. The data extension unit 23 generates data by performing image processing (trimming and rotation) on four randomly selected teacher data, as one extended data ED1, and stores the data in the extended data DB 272 of the storage unit 27.

[0042] The learning unit 26 creates a learning model for sorting both cast materials and wrought materials from a plurality of objects OB by performing deep learning using a plurality of teacher data stored in the teacher data DB 271 and a plurality of extended data stored in the extended data DB 272. The created learning model is stored in the model DB 273 and is used for sorting by the sorting device of the objects OB.

[0043] 2. How to generate augmented training data: Fig. 5 is a flowchart of the extended teacher data generation method. In the extended teacher data generation flow shown in Fig. 5, first, the belt conveyor 10 transports a plurality of objects OB (step S1). The photographing unit 30 photographs each object OB transported by the belt conveyor 10 (step S2). The position acquisition unit 25 acquires position information of the object OB by performing image recognition on the image of the object OB acquired by the photographing unit 30 (step S3). The position acquisition unit 25 acquires position information of the object OB by extracting a bounding box for the image in which the object OB appears.

[0044] The material acquisition unit 40 acquires material information of the object OB conveyed by the belt conveyor 10 by laser induced breakdown spectroscopy (step S4). The material acquisition unit 40 adjusts the focus and irradiation timing of the laser LS using the position information of the object OB and the conveying speed of the belt conveyor 10. The material acquisition unit 40 identifies the material of the object OB by using the emission spectral line intensity obtained from the object OB.

[0045] The data generating unit 24 generates teacher data that associates the acquired material information of the object OB, the image of the object OB acquired by the photographing unit 30, and the position information of the object OB acquired by the position acquiring unit 25, and stores the teacher data in the teacher data DB 271 (step S5). The data expanding unit 23 generates a composite image IM by combining images included in multiple teacher data randomly selected from the multiple teacher data. ED (Step S6). The data extension unit 23 generates extended data by selecting many pieces of teacher data associated with materials having a small ratio among the materials included in the multiple teacher data stored in the teacher data DB 271. When the multiple pieces of extended data are generated, the extended teacher data generation flow ends.

[0046] 3.Evaluation of the learning model using training data and augmented training data: A scrap sorting evaluation was performed using a learning model created by the learning unit 26 using the teacher data and extended teacher data (extended data). The data extension unit 23 generated one or more extended data using a total of 49 teacher data, including 26 teacher data whose material was cast material and 23 teacher data whose material was wrought material.

[0047] Fig. 6 is a table showing the relationship between the number of teacher data and extended teacher data and the non-detection rate of scrap. Fig. 6 shows the non-detection rate (%) of cast materials and wrought materials when sorting multiple scraps when the total number of teacher data and extended data used to create the learning model is 50, 100, and 200. As shown in Fig. 6, as the total number of teacher data and extended data used for learning increases from 50 to 200, the non-detection rate of cast materials and wrought materials decreases.

[0048] FIG. 7 is a table showing the relationship between the ratio of materials included in the teacher data and the extended teacher data and the undetected rate. FIG. 7 shows the undetected rate of scrap of cast material, which changes according to the ratio of data in which the material of the object OB included in all the teacher data and the extended data used to create the learning model is cast material. As shown in FIG. 7, as the ratio of teacher data whose material is cast material included in the total of all the teacher data and the extended data increases, the undetected rate of scrap of cast material by the learning model decreases. Note that the ratio of cast material included in the total of the teacher data and the extended data of 2% represents the ratio of data of cast material among all materials included in the total of the teacher data and the extended data. For example, when the number of teacher data is 200 and the number of extended data is 200, the number of original teacher data included in the total of the teacher data and the extended data is 1000 (200 + 200 × 4), and the ratio of 2% represents 20 out of 1000 original teacher data.

[0049] FIG. 8 is a table showing the generation time of teacher data in an embodiment and a comparative example. FIG. 8 shows the generation time of an embodiment in which teacher data is automatically created using the learning model creation system 100 of this embodiment, and the generation time of a comparative example in which teacher data is created manually as has been conventionally done. The generation time shown in FIG. 8 is the time it takes to generate teacher data from scrap waste, which is 2000 objects OB. In the embodiment, when the interval between the objects OB is 300 mm and the conveying speed of the belt conveyor 10 is 5 m / min, the generation time calculated by the following formula (1) is 2 h. 300mm / (5m / min)×2000 pieces=2h...(1)

[0050] In the comparative example, the time required for an operator to obtain the material of each object OB is 5 seconds, the time required for photographing each object OB is 5 seconds, and the time required for annotating the material information and the image of each object OB is 10 seconds. In this case, the generation time in the comparative example calculated by the following formula (2) is about 11 hours. (5s+5s+10s)×2000 pieces≒11h...(2)

[0051] As described above, in the learning model creation system 100 of this embodiment, the material acquisition unit 40 automatically acquires material information of the object OB using the position information of the object OB acquired by the position acquisition unit 25. Then, the data generation unit 24 generates teacher data in which the image of each object OB acquired by the photographing unit 30, the position information acquired by the position acquisition unit 25, and the material information acquired by the material acquisition unit 40 are associated with each other. That is, in the learning model creation system 100 of this embodiment, the data generation unit 24 generates teacher data in which the image, position information, and material information of the object OB automatically acquired by the photographing unit 30, the position acquisition unit 25, and the material acquisition unit 40 are associated with each other. Since a huge amount of teacher data is required to create a learning model, the time required to create a learning model (learning process) can be reduced by automating the generation of teacher data as in this embodiment. Furthermore, automatic sorting using a learning model can realize more accurate sorting than sorting by an operator, and can realize high-speed sorting. In other words, by using the learning model creation system 100 of this embodiment, it is possible to achieve labor savings, high accuracy, and increased throughput in the processes from machine learning to automatic selection of objects OB.

[0052] In addition, the learning model creation system 100 of this embodiment performs image processing on an image of the object OB included in the original teacher data, and generates data in which the image of the object OB after the image processing is associated with the position information and material information included in the original teacher data. That is, the data extension unit 23 performs image processing on the original teacher data to generate extended data, so that extended teacher data that has been modified can be easily generated, and a large amount of extended data can be generated without any effort. As a result, a learning model is created using more extended data than the teacher data, so that the detection rate of the object OB in automatic selection using the learning model can be improved, as shown in FIG. 6.

[0053] The data extension unit 23 of this embodiment includes a data extension unit 23 that uses the teacher data generated by the data generation unit 24 to generate extended data in which multiple teacher data are associated as one data. The data extension unit 23 performs image processing of trimming and rotation on the image of the object OB included in the original teacher data to generate the extended data. As shown in FIG. 3, the data extension unit 23 pastes each of the trimmed images of the images IM1 to IM4 in a randomly determined area of ​​the four divided areas AR1 to AR4 without overlapping. As a result, extended data that is greater than the number of teacher data is generated, and the generated extended data includes multiple original teacher data. Therefore, the learning model created using the extended data of this embodiment can achieve an improved processing amount compared to the learning model created using the teacher data, even if multiple objects OB are transported by the belt conveyor 10 during automatic sorting.

[0054] In addition, the data extension unit 23 of this embodiment selects two teacher data of cast material and two teacher data of wrought material to be associated with the extended data so that the number of objects OB whose material is cast material is approximately equal to the number of objects OB whose material is wrought material. This reduces bias in the material contained in the original teacher data contained in all extended data even if the number of data containing objects OB of a specific material as teacher data is small. As a result, as shown in FIG. 7, the detection rate of objects to be selected can be improved in automatic selection of objects OB using a learning model created using teacher data and extended data.

[0055] The learning model creation system 100 of this embodiment also includes a belt conveyor 10 that transports a plurality of objects OB. Therefore, while the positions of the photographing unit 30 and the position acquiring unit 25 are fixed, the position information and material information of the object OB are automatically acquired.

[0056] Furthermore, the belt conveyor 10 of the present embodiment conveys a plurality of objects OB in a line on the belt 11 along the conveying direction so that one object OB is included in one image acquired by the photographing unit 30. Therefore, the data generating unit 24 can easily generate teacher data in which images of the objects OB are associated with each other.

[0057] Furthermore, the data generating unit 24 of this embodiment generates training data including an image and position information of one object OB acquired by the photographing unit 30, and material information acquired by the material acquiring unit 40. Therefore, training data associated with an image in which multiple objects OB are captured is not used for machine learning, so that the created learning model can be made more accurate.

[0058] In addition, since the material acquisition unit 40 of this embodiment is a laser-induced breakdown spectrometer, the time required to identify the material of the object OB can be reduced. In addition, the laser-induced breakdown spectrometer of this embodiment automatically adjusts the focal distance to the object OB using height information included in the position information to the object OB on the belt 11 acquired by the position acquisition unit 25. This automates the acquisition of position information and material information of the object OB transported by the belt conveyor 10, thereby realizing labor savings when generating training data.

[0059] Moreover, the object OB to be sorted in this embodiment is scrap waste. The shape of the waste retains the characteristics of the original shape, and the specific parts that are the source of the waste are made of specific materials, so there is often a correlation between the shape and material of the waste. Therefore, in this embodiment, by recognizing in advance what kind of waste the object OB is, teacher data is created that associates an image showing the shape of the object OB with material information of the object OB. As a result, in this embodiment, the learning model using the teacher data can improve the amount of sorting processing when automatically sorting materials based on images of the object OB.

[0060] In addition, the photographing unit 30 of this embodiment photographs each object OB with a resolution that captures the object OB in an area of ​​10 pixels x 10 pixels or more. That is, since the photographing unit 30 can obtain an image of the object OB having a predetermined size or more, the data generating unit 24 generates teacher data for creating a learning model with higher accuracy.

[0061] Moreover, the position acquisition unit 25 of this embodiment performs image processing (grayscaling, etc.) on the image in which the object OB appears, and acquires position information of the object OB in the image. By identifying the position of the object OB in the image, the data generation unit 24 can generate teacher data for creating a learning model that performs highly accurate selection.

[0062] <Modifications of the above embodiment> The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit of the present invention. For example, the following modifications are also possible. In the above-described embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. The present invention is not limited to the above-described embodiment, and can be embodied in various forms without departing from the spirit of the present invention.

[0063] [Variation 1] The learning model creation system 100 of the above embodiment is an example of a system including a teacher data generation device that generates teacher data, and the configuration of the teacher data generation device and the control performed by the teacher data generation device can be modified in various ways. For example, the teacher data generation device does not need to include the learning unit 26, the monitor 28, and the data extension unit 23. The teacher data generation device only needs to include the photographing unit 30, the position acquisition unit 25 that acquires position information of the object OB, the material acquisition unit 40 that acquires material information of the object OB, and the data generation unit 24 that generates teacher data. The teacher data generation device does not need to generate extension data, and does not need to create a learning model by the learning unit 26. The learning model may be created by another device using the teacher data generated by the teacher data generation device.

[0064] In the above embodiment, the data extension unit 23 generates one extended data ED1 by performing image processing on the image of the object OB included in the four selected teacher data, but the generated extended data can be modified. For example, the data extension unit 23 may generate one extended data by randomly rotating the image of the object OB included in one teacher data. The number of original teacher data included in the extended data may be three or less, or may be five or more. In addition, the data extension unit 23 may paste the image of the object OB included in the teacher data onto the background image IMB without rotating it, or may mix rotated images and non-rotated images. The data extension unit 23 may divide the areas AR1 to AR4 into which the background image IMB is divided into different sizes and shapes. The image of the object OB included in the original teacher data pasted onto the background image IMB may be enlarged or reduced. In the extended data, the area ID shown in FIG. 4 may be stored as position information.

[0065] The data extension unit 23 in the above embodiment selected two teacher data items associated with the extended data ED1 so that the number of objects OB made of cast material is approximately equal to the number of objects OB made of wrought material, but the teacher data may be selected without considering the ratio of the number of items according to material. It is preferable that the data extension unit 23 selects original teacher data items associated with the extended data so that the ratio of the number of items according to material is the same. For example, when the number of types of materials included in the teacher data is Nm, the number of teacher data items associated with one extended data item is Nd, and the number of extended data items to be generated is Ne, it is preferable that the number of original teacher data items for each material included in all the extended data items is close to Nd×Ne / Nm (items). In this embodiment, "approximately equal" refers to a value within a range of ±20% from the most preferable value.

[0066] [Variation 2] The multiple objects OB transported for generating the teacher data do not have to be arranged in a line on the belt conveyor 10. In this case, the data generating unit 24 can generate teacher data corresponding to one object OB by not generating teacher data when multiple objects OB are captured in one image acquired by the photographing unit 30. The belt conveyor 10 may be provided with a guide for arranging the objects OB in a line at a predetermined interval.

[0067] Furthermore, the data generating unit 24 may generate teacher data corresponding to the object OB when the material acquired by the material acquiring unit 40 satisfies a preset material criterion of being a cast material or a wrought material, and may not generate teacher data corresponding to the object OB when the material does not satisfy the material criterion. In this modification, whether or not to generate teacher data is determined according to a certain material criterion, and therefore teacher data associated with an unclear material as material information is not generated. In other words, the data generating unit 24 can create highly accurate teacher data.

[0068] The material acquisition unit 40 in the above embodiment is a laser-induced breakdown spectrometer, but may be a hyperspectral camera, or may identify the material of the object OB by X-ray analysis. The photographing unit 30 in the above embodiment is a single stereo camera, but may be configured with multiple cameras, and the number and positions of the cameras arranged as the photographing unit can be modified. In addition, the resolution of the photographing unit 30 does not have to be 10×10 pixels or more, which is sufficient to photograph the object OB.

[0069] Although the present aspect has been described above based on the embodiment and modified examples, the above-mentioned embodiment of the aspect is intended to facilitate understanding of the present aspect and does not limit the present aspect. The present aspect may be modified or improved without departing from the spirit and scope of the claims, and equivalents are included in the present aspect. Furthermore, if a technical feature is not described as essential in this specification, it may be deleted as appropriate. [Explanation of symbols]

[0070] 10...Conveyor belt 11. Belt 20...Model making equipment 22...Control section 23...Data extension unit (teacher data extension unit) 24...Data generation unit (teacher data generation unit) 25...Position acquisition section 26…Learning Department 27...Storage section 28…Monitor 30…Photography Department 40…Material acquisition section 100...Learning model creation system (teaching data generation device) 271…Teacher data DB 272…Extended Data DB 273…Model DB AR1~AR4…area ED1...Expanded data IM1~IM4, IM1a~IM4a...Images of the object IMB…Background image IM ED…Composite image LS...Laser OB…Object P1A, P1B, P2A, P1Aa, P1Ba, P2Aa...Feature points SM1~SM4…Sample

Claims

1. A teacher data generation device, An image capturing unit for capturing an image of an object to be selected; a position acquisition unit that acquires position information of the object using an image acquired by the photographing unit; a material acquisition unit that acquires material information of the object by a laser induced breakdown spectrometer that measures the material of the object using position information of the object; a teacher data generating unit that generates teacher data in which the image of the object, the position information, and the material information are associated with each other; A teacher data extension unit that generates one or more extended teacher data using the teacher data generated by the teacher data generation unit; Equipped with The teacher data extension unit of the teacher data generation device performs image processing on an image of the object contained in the original teacher data, and generates the extended teacher data by matching the image of the object after the image processing with the position information and the material information contained in the original teacher data.

2. The teacher data generating device according to claim 1, The teacher data extension unit performs the image processing as follows: Select a plurality of the original teacher data; A teacher data generation device that performs a process of generating a cropped image by cutting out a specified area including the object for each image of the object included in the selected teacher data, rotating the cropped image at a random angle, and generating a composite image by pasting the cropped image at a randomly determined position on a single background image without overlapping.

3. The teacher data generating device according to claim 2, A teacher data generating device in which the teacher data extension unit generates the extended teacher data so that the number of objects associated with all of the generated extended teacher data is approximately the same for each type of material of the objects.

4. The teacher data generating device according to any one of claims 1 to 3, further comprising: A teacher data generation device comprising a transport unit that transports the object.

5. The teacher data generating device according to claim 4, A teacher data generation device in which the transport unit transports multiple selection objects in a line along a transport direction so that one object is included in one image acquired by the photographing unit.

6. The teacher data generating device according to claim 5, The teacher data generation unit generates the teacher data that corresponds, for one of the objects, an image of the object, the position information, and the material information.

7. The teacher data generating device according to any one of claims 1 to 6, The subject is metal scrap waste, a teacher data generation device.

8. The teacher data generating device according to claim 7, The photographing unit photographs the scrap at a resolution that captures the scrap in an area of ​​10 pixels x 10 pixels or more.

9. The teacher data generating device according to any one of claims 1 to 8, The position acquisition unit of the teacher data generation device identifies the position of the object appearing in the image by performing image recognition on the image acquired by the photographing unit.

10. The teacher data generating device according to any one of claims 1 to 9, The teacher data generation unit is When the material information acquired by the material acquisition unit satisfies a preset material criterion, the teacher data is generated using the object that satisfies the material criterion; A teacher data generating device that, if the material information does not satisfy the material criterion, does not generate the teacher data associated with the object that does not satisfy the material criterion.

11. A teacher data generation method, comprising: An imaging step of acquiring an image of the object to be selected; a position acquisition step of acquiring position information of the object using the acquired image; a material acquisition step of acquiring material information of the object by a laser induced breakdown spectrometer that measures the material of the object using position information of the object; a training data generating step of generating training data in which the image of the object, the position information, and the material information are associated with each other; a teacher data extension step of generating one or more extended teacher data using the teacher data generated by the teacher data generation step; Equipped with A teacher data generation method in which the teacher data extension process performs image processing on an image of the object contained in the original teacher data, and generates the extended teacher data by matching the image of the object after the image processing with the position information and material information contained in the original teacher data.

12. A computer program for causing a computer to generate teacher data, A photographing function for acquiring an image of the object to be sorted; A position acquisition function that acquires position information of the object using the acquired image; a material acquisition function that acquires material information of the object by a laser induced breakdown spectrometer that measures the material of the object using position information of the object; A training data generating function that generates training data in which the image of the object, the position information, and the material information are associated with each other; A teacher data extension function that generates one or more extended teacher data using the teacher data generated by the teacher data generation function; Realize this, The teacher data expansion function is a computer program that performs image processing on an image of the object contained in the original teacher data, and generates the extended teacher data by matching the image of the object after the image processing with the position information and material information contained in the original teacher data.

Images