Image generation system, image generation method, and program

Abstract

The objective of the present disclosure is to generate an image suitable for machine learning. An image generation system (5) comprises a first image acquisition unit (51), a second image acquisition unit (52), and an image processing unit (53). The image processing unit (53) carries out an image conversion process (processes 72 and 73) and a superimposition process (74). The image conversion process is a process for generating a converted image (P3), which is an image obtained by processing a prescribed extraction site of a first object on the basis of a first image (P1). The image conversion process includes a process for dividing the extraction site into a plurality of divided parts, and a process for changing at least one of the gradation, position, size, and orientation of at least one divided part among the plurality of divided parts.

Description

Technical Field

[0001] The present disclosure generally relates to an image generation system, an image generation method, and a program, and more particularly to an image generation system, an image generation method, and a program that generate an image by superimposing a plurality of images.

Background Art

[0002] The image display device (image generation system) described in Patent Document 1 includes an image specifying means, an image processing means, an image synthesizing means, and an image displaying means. The image specifying means specifies an original image to be processed. The image processing means processes the original image to generate a processed image. The image synthesizing means generates an entire image by replacing a partial region of the image with a corresponding partial region of the processed image. The image displaying means displays all or a part of the image including the boundary of the entire image.

[0003] Thus, with the image display device described in Patent Document 1, a new image can be generated from the original image. However, it has been difficult to generate an image suitable for machine learning with this image display device.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] An object of the present disclosure is to provide an image generation system, an image generation method, and a program that can generate an image suitable for machine learning.

[0006] An image generation system according to an aspect of the present disclosure includes a first image acquisition unit, a second image acquisition unit, and an image processing unit. The first image acquisition unit acquires a first image obtained by imaging a first object. The second image acquisition unit acquires a second image obtained by imaging a second object. The image processing unit performs an image conversion process and a superimposition process. The image conversion process is a process of generating a converted image, which is an image obtained by processing a predetermined extraction part of the first object based on the first image. The superimposition process is a process of generating a superimposed image obtained by superimposing the converted image on the second image. The image conversion process includes a process of dividing the extraction part into a plurality of divided parts and a process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts.

[0007] An image generation method according to an aspect of the present disclosure includes a first image acquisition process, a second image acquisition process, an image conversion process, and a superimposition process. In the first image acquisition process, a first image obtained by imaging a first object is acquired. In the second image acquisition process, a second image obtained by imaging a second object is acquired. The image conversion process is a process of generating a converted image, which is an image obtained by processing a predetermined extraction part of the first object based on the first image. The superimposition process is a process of generating a superimposed image obtained by superimposing the converted image on the second image. The image conversion process includes a process of dividing the extraction part into a plurality of divided parts and a process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts.

[0008] A program according to an aspect of the present disclosure is a program for causing one or more processors of a computer system to execute the image generation method.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0010] (Embodiment) Hereinafter, the image generation system 5 according to the embodiment will be described with reference to the drawings. However, the following embodiment is only one of various embodiments of the present disclosure. The following embodiment can be variously modified according to the design and the like as long as the object of the present disclosure can be achieved. Also, each drawing described in the following embodiment is a schematic diagram, and the ratio of the size and thickness of each component in the drawing does not necessarily reflect the actual dimensional ratio.

[0011] (Overview) As shown in FIG. 1, the image generation system 5 according to the present embodiment creates a superimposed image P4 from a first image P1 and a second image P2. The superimposed image P4 is used as learning data for generating a learned model 82 related to an object (for example, an equivalent of the second object 2 (see FIG. 3)). That is, the superimposed image P4 is learning data used for generating a model by machine learning.

[0012] As used in this disclosure, a "model" is a program that, when input information regarding an object to be recognized (the object) is input, recognizes the state of the object to be recognized and outputs a recognition result. A "trained model" refers to a model for which machine learning using training data has been completed. Also, "training data (training dataset)" is a dataset that combines input information (an image) input to the model and a label assigned to the input information, and is so-called teacher data. That is, in the present embodiment, the trained model 82 is a model for which machine learning by supervised learning has been completed.

[0013] The trained model 82 referred to here may include, for example, a model using a neural network or a model generated by deep learning (deep neural network) using a multi-layer neural network. The neural network may include, for example, a CNN (Convolutional Neural Network) or a BNN (Bayesian Neural Network). The trained model 82 is realized by implementing a trained neural network in an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array). The trained model 82 is not limited to a model generated by deep learning. The trained model 82 may also be a model generated by a support vector machine or a decision tree.

[0014] In the present embodiment, as an example, the object to be recognized is a welded object. FIG. 5 is an example of a superimposed image P4 (training data), and the object 4 is shown in the superimposed image P4. Similar to the object to be recognized, the object 4 is also a welded object. The object 4 includes a first metal plate 41, a second metal plate 42, and a bead 43.

[0015] The bead 43 is formed at the boundary B1 (welding location) between the first metal plate 41 and the second metal plate 42 when two or more base materials (here, the first metal plate 41 and the second metal plate 42) are welded using a metal welding material. The size and shape of the bead 43 mainly depend on the welding material. Further, the object 4 includes a defective part 44. Hereinafter, the location where the defective part 44 exists is referred to as the defective occurrence location E4. In FIG. 5, the defective occurrence location E4 is included in the second metal plate 42.

[0016] When an image (inspection image P5) of an object to be recognized (object) is input, the learned model 82 recognizes the state of the object and outputs a recognition result. Specifically, the learned model 82 outputs, as the recognition result, whether the object is a non-defective product or a defective product, and when it is a defective product, outputs the type of the defective product. That is, the learned model 82 is used for inspecting whether the object is good or bad, in other words, for welding appearance inspection to check whether the welding was performed correctly.

[0017] Whether the object is a non-defective product or not is determined, for example, by whether the length of the bead, the height of the bead, the rising angle of the bead, the throat thickness of the bead, the reinforcement of the bead, and the displacement of the welding location of the bead (including the displacement of the start end of the bead) are within the allowable range. For example, if even one of the conditions listed above is not within the allowable range, the object is determined to be a defective product. Also, whether the object is a non-defective product or not is determined, for example, based on the presence or absence of defective parts such as undercut of the object, pits on the bead, spatter on the bead, and protrusions on the bead. For example, if even one of the defective parts listed above occurs, the object is determined to be a defective product.

[0018] By the way, in order to perform machine learning of the model, it is necessary to prepare a large number of image data including defective products as learning data. However, when the frequency of defective products occurring in the production line of the object is low, the learning data necessary to generate a well-trained model 82 with a high recognition rate is likely to be insufficient. Therefore, it is conceivable to perform data augmentation processing on the learning data (the first image P1 and the second image P2) obtained by actually imaging the object, increase the number of learning data, and perform machine learning of the model. The data augmentation process refers to a process of increasing the number of learning data by performing processes such as synthesis, translation, enlargement / reduction, rotation, inversion, or addition of noise on the learning data. In the present embodiment, by performing the data augmentation process, at least one (a large number in the inventor's assumption) superimposed image P4 is generated, and at least one superimposed image P4 is used as learning data, respectively. Further, in the present embodiment, a superimposed image P4 is generated by performing a process of superimposing (synthesizing) these plurality of original images on a plurality of original images (a plurality of learning data) before being processed by the image generation system 5.

[0019] It is not essential that the plurality of original images (the first image P1 and the second image P2) used to generate the superimposed image P4 be used as learning data for generating the well-trained model 82. That is, the learning data for generating the well-trained model 82 may include only a plurality of superimposed images P4, or may include the first image P1, the second image P2, etc., which are not images generated by the image generation system 5, in addition to at least one superimposed image P4. That is, the learning data for generating the well-trained model 82 may or may not include the original images before being processed by the image generation system 5. Further, the learning data for generating the well-trained model 82 may include images generated by a system other than the image generation system 5.

[0020] As shown in FIG. 1, the image generation system 5 of the present embodiment includes a first image acquisition unit 51, a second image acquisition unit 52, and an image processing unit 53. The first image acquisition unit 51 acquires a first image P1 obtained by imaging a first object 1. The second image acquisition unit 52 acquires a second image P2 obtained by imaging a second object 2. The image processing unit 53 performs an image conversion process (processes 72 and 73) and a superimposition process 74. The image conversion process is a process of generating a converted image P3, which is an image obtained by processing a predetermined extraction part 140 of the first object 1 based on the first image P1. The superimposition process 74 is a process of generating a superimposed image P4 by superimposing the converted image P3 on the second image P2. The image conversion process includes a process of dividing the extraction part 140 into a plurality of divided parts 141, and a process of changing at least one of the gradation, position, size, and orientation of each of at least one of the plurality of divided parts 141.

[0021] According to the present embodiment, an image (superimposed image P4) suitable for machine learning can be generated. In particular, before generating the superimposed image P4 by superimposing the converted image P3 on the second image P2, a process of dividing the extraction part 140 into a plurality of divided parts 141 is performed, and further, a process of changing at least one of the gradation, position, size, and orientation of each of at least one of the plurality of divided parts 141 (hereinafter referred to as a conversion process) is performed. That is, the conversion process is performed for each of the divided parts 141. As a result, compared with the case where the extraction part 140 is not divided, it becomes easier to determine a parameter (setting information 81) for determining the conversion process, and it becomes easier to generate a superimposed image P4 desired by a user or the like. Therefore, it is possible to improve the accuracy of machine learning using the superimposed image P4.

[0022] Also, functions similar to those of the image generation system 5 can be embodied in an image generation method. The image generation method of the present embodiment includes a first image acquisition process, a second image acquisition process, image conversion processes (processes 72 and 73), and an overlapping process 74. In the first image acquisition process, a first image P1 obtained by imaging a first object 1 is acquired. In the second image acquisition process, a second image P2 obtained by imaging a second object 2 is acquired. The image conversion process is a process of generating a converted image P3, which is an image obtained by processing a predetermined extraction part 140 of the first object 1 based on the first image P1. The overlapping process 74 is a process of generating an overlapping image P4 by overlapping the converted image P3 on the second image P2. The image conversion process includes a process of dividing the extraction part 140 into a plurality of divided parts 141, and a process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts 141.

[0023] Also, the image generation method preferably further includes a setting information generation process. The setting information generation process is a process of generating setting information 81 (see FIG. 1) regarding the process of generating the overlapping image P4 from the first image P1 and the second image P2.

[0024] Also, the image generation method is used on a computer system (image generation system 5). That is, the image generation method can be embodied by a program. The program according to the present embodiment is a program for causing one or more processors of a computer system to execute the image generation method according to the present embodiment. The program may be recorded on a non-transitory recording medium readable by the computer system.

[0025] (Details) Hereinafter, the image generation system 5 according to the present embodiment will be described in more detail.

[0026] (1) Overall configuration The image generation system 5 shown in FIG. 1 includes a computer system having one or more processors and a memory. At least some of the functions of the image generation system 5 are realized by the processor of the computer system executing a program recorded in the memory of the computer system. The program may be recorded in the memory, may be provided through a telecommunication line such as the Internet, or may be provided by being recorded on a non-transitory recording medium such as a memory card.

[0027] The image generation system 5 may be installed in a factory that is a welding site, or at least a part of the configuration of the image generation system 5 may be installed outside the factory (for example, an office provided at a location different from the factory).

[0028] As described above, the image generation system 5 has a function of performing data augmentation processing on an original image (learning data) to increase the number of learning data. Hereinafter, a person who uses the image generation system 5 may be simply referred to as a "user". The user is, for example, an operator who monitors a manufacturing process such as a welding process in a factory, or a person in charge of management.

[0029] Note that in this embodiment, as described above, the learned model 82 is generated by machine learning. The learned model 82 may be implemented as any type of artificial intelligence or system. Here, as an example, the machine learning algorithm is a neural network. However, the machine learning algorithm is not limited to a neural network, and may be, for example, XGB (eXtreme Gradient Boosting) regression, Random Forest, decision tree, Logistic Regression, Support Vector Machine (SVM), Naive Bayes classifier, or k-nearest neighbors, etc. Further, the machine learning algorithm may be, for example, Gaussian Mixture Model (GMM) or k-means clustering, etc.

[0030] Also, the learned model 82 is not limited to machine learning that classifies inspection images into two classes of good products and defective products, and may be generated by machine learning that classifies inspection images into multiple classes such as good products, pits, sputters, protrusions, melt drops, undercuts, etc. Pits, sputters, protrusions, melt drops, and undercuts are each types of defects. Also, the learned model 82 may be generated by machine learning that performs object detection or segmentation to detect the position and type of defect in the inspection image.

[0031] Also, as an example in the embodiment, the learning method is supervised learning. However, the learning method is not limited to supervised learning, and may be unsupervised learning or reinforcement learning.

[0032] Also, in the example shown in FIG. 1, the image generation system 5 includes a storage unit 63 for storing (memorizing) learning data. The storage unit 63 includes a rewritable non-volatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit 63 may be a memory built into the image generation system 5. However, the storage unit 63 may be provided outside the image generation system 5.

[0033] As shown in FIG. 1, the image generation system 5 includes a first image acquisition unit 51, a second image acquisition unit 52, an image processing unit 53, a setting information input unit 54, a user interface 55, a setting information generation unit 56, a display output unit 57, a display device 58. Further, the image generation system 5 includes an image output unit 59, a learning unit 60, a determination unit 61, an inspection image acquisition unit 62, and a storage unit 63.

[0034] At least the user interface 55, the display device 58, and the storage unit 63 are physical components. On the other hand, the first image acquisition unit 51, the second image acquisition unit 52, the image processing unit 53, the setting information input unit 54, the setting information generation unit 56, the display output unit 57, the image output unit 59, the learning unit 60, the determination unit 61, and the inspection image acquisition unit 62 merely indicate functions realized by one or more processors of the image generation system 5, and do not necessarily indicate physical components.

[0035] (2) First Image Acquisition Unit The first image acquisition unit 51 acquires a first image P1. As shown in FIG. 2, the first image P1 is an image obtained by imaging the first object 1. The first image acquisition unit 51 may acquire the first image P1 from a device outside the image generation system 5, or may acquire the first image P1 from the storage unit 63 of the image generation system 5. In the example shown in FIG. 1, the first image acquisition unit 51 acquires the first image P1 from a device outside the image generation system 5. For example, the first image acquisition unit 51 acquires the first image P1 from a computer server.

[0036] Further, the imaging device that images the first object 1 to generate the first image P1 may be provided in the image generation system 5 or may be provided outside the image generation system 5.

[0037] The first object 1 is a defective product. A defective product is an article in which a defect has occurred. What state of the article is regarded as the state in which a defect has occurred may be appropriately determined by the user of the image generation system 5.

[0038] The first image P1 is an image (defective product image) that images at least the defect occurrence location E1 of the first object 1. However, the range shown in the first image P1 is not limited to only the defect occurrence location E1, and may also include locations other than the defect occurrence location E1 of the first object 1. In the example shown in FIG. 2, the range shown in the first image P1 is the entire first object 1.

[0039] The first object 1 includes a first metal plate 11, a second metal plate 12, and a bead 13. Since the configurations of the first metal plate 11, the second metal plate 12, and the bead 13 are the same as those of the first metal plate 41, the second metal plate 42, and the bead 43 of the object 4 described above, the description thereof is omitted. However, in the first object 1, a defective part 14 having a shape different from that of the defective part 44 exists at a location different from the location (defect occurrence location E4) where the defective part 44 is provided in the object 4. More specifically, the defective part 14 exists in the first metal plate 11.

[0040] (3) Second Image Acquisition Unit The second image acquisition unit 52 acquires a second image P2. As shown in FIG. 3, the second image P2 is an image obtained by imaging the second object 2. The second image acquisition unit 52 may acquire the second image P2 from a device outside the image generation system 5 or may acquire the second image P2 from the storage unit 63 of the image generation system 5. In the example shown in FIG. 1, the second image acquisition unit 52 acquires the second image P2 from a device outside the image generation system 5. For example, the second image acquisition unit 52 acquires the second image P2 from a computer server.

[0041] Also, the imaging device that images the second object 2 to generate the second image P2 may be provided in the image generation system 5 or may be provided outside the image generation system 5.

[0042] The second object 2 is a non-defective product. A non-defective product is an article in which no defect has occurred. In the example shown in the figure 3 In the example shown, the range shown in the second image P2 is the entire second object 2. However, the range shown in the second image P2 may be a part of the second object 2.

[0043] The second object 2 includes a first metal plate 21, a second metal plate 22, and a bead 23. Since the configurations of the first metal plate 21, the second metal plate 22, and the bead 23 are the same as those of the first metal plate 41, the second metal plate 42, and the bead 43 of the object 4 described above, the description thereof is omitted. However, the second object 2 does not have a configuration corresponding to the defective part 44.

[0044] (4) The first image and the second image The first image P1 is, for example, a distance image including information on coordinates in the depth direction (the direction from the imaging device toward the first object 1). The second image P2 is, for example, a distance image including information on coordinates in the depth direction (the direction from the imaging device toward the second object 2). The information on the coordinates in the depth direction is represented by, for example, gradation. Specifically, in the distance image, the greater the density of the target point, the more the target point is located in the back. However, conversely, in the distance image, the smaller the density of the target point, the more the target point is located in the back may be represented.

[0045] The imaging device for generating a distance image is a distance image sensor such as a line sensor camera. By the imaging device, a plurality of objects are sequentially imaged, and a plurality of images are generated. From among the plurality of images generated by the imaging device, for example, according to a user's instruction, the first image P1 and the second image P2 are selected. The image generation system 5 preferably includes an operation unit that receives an instruction regarding the selection. For example, the user interface 55 may be used as the operation unit.

[0046] (5) Image processing unit The image processing unit 53 is realized by, for example, a DSP (Digital Signal Processor) or an FPGA (Field-Programmable Gate Array). The image processing unit 53 performs an extraction process 71, an image conversion process (processes 72, 73), and a superimposition process 74. The image processing unit 53 performs the extraction process 71, the image conversion process (processes 72, 73), and the superimposition process 74 based on the setting information 81. As will be described later, the setting information 81 may be input by the user or may be automatically generated by the setting information generation unit 56.

[0047] In the present embodiment, the image generated by the image conversion process (processes 72, 73) is referred to as a converted image P3. Also, the image generated during the intermediate process of the image conversion process (processes 72, 73) is also referred to as the converted image P3. Further, the image generated by the superimposition process 74 is referred to as a superimposed image P4.

[0048] The extraction part 140 is a part of the first object 1. The extraction process 71 is a process of extracting the extraction part 140 from among the first object 1 shown in the first image P1 (see FIG. 2). That is, as shown in FIG. 4A, an extraction image from which the extraction part 140 is extracted is generated from the first image P1.

[0049] In this embodiment, the extraction site 140 coincides with the defective site 14. The defective site 14 coincides with the location E1 where the defect occurs. That is, the first image P1 is an image capturing the location E1 of the defect of the first object 1, the second image P2 is an image capturing the second object 2 as a non-defective product, and the extraction site 140 includes at least a part (in this embodiment, all) of the location E1 where the defect occurs.

[0050] The process 72 includes a deformation process and a division process. Either the deformation process or the division process may be performed first. In this embodiment, it will be described assuming that at least a part of the deformation process is performed first and then the division process is performed.

[0051] The deformation process is a process of deforming the extraction site 140 so as to bend it. More specifically, the deformation process is a process of deforming the extraction site 140 along the shape of the second object 2 in the second image P2. Also, the deformation process may include a process of stretching or compressing the extraction site 140 in a predetermined direction. The deformation process may include a process of inverting (mirror-inverting) the extraction site 140. By performing the deformation process on the extraction image, for example, a transformed image P3 as shown in FIG. 4B is generated. And in the superimposition process 74, the extraction site 140 is arranged along the shape of the second object 2.

[0052] By deforming the extraction site 140 along the shape of the second object 2, for example, a linear extraction site 140 may become curved, or conversely, a curved extraction site 140 may become linear. Alternatively, a curved extraction site 140 may become a curved extraction site 140 having a shape different from that of the extraction site 140 before deformation. Deformations other than those listed here may be made to the extraction site 140.

[0053] Process 72 may include a process of extracting at least a part of the contour or edge of the second object 2 in the second image P2. For example, a portion where the luminance changes more than a threshold value in the second image P2 may be extracted as a contour or an edge. Then, in the deformation process, the image processing unit 53 may deform the extraction site 140 so as to bend along the contour or edge. That is, in the image conversion process, deforming the extraction site 140 along the shape of the second object 2 in the second image P2 may mean deforming the extraction site 140 along at least a part of the contour or edge of the second object 2 in the second image P2. In this case, in the superimposition process 74, the extraction site 140 is arranged along at least a part of the contour or edge of the second object 2. In FIG. 4B, the extraction site 140 is deformed along the contour or edge of the upper right to lower right portion of the bead 43 of the second object 2 shown in FIG. 5.

[0054] The splitting process is a process of splitting the extraction site 140 into a plurality of split parts 141. After the deformation process is performed on the extraction image, the splitting process is performed, whereby, for example, a converted image P3 as shown in FIG. 4C is generated. Thus, the image conversion process includes a process of splitting the extraction site 140 into a plurality of split parts 141. In FIG. 4C, the boundaries of each split part 141 are represented by rectangular frame lines. FIG. 4D is an enlarged view of a part of FIG. 4C.

[0055] Process 73 includes a process of changing at least one of the gradation of the extraction site 140, the position of the extraction site 140, the size of the extraction site 140, and the orientation of the extraction site 140. That is, the image conversion process includes a process of changing at least one of the gradation of the extraction site 140, the position of the extraction site 140, the size of the extraction site 140, and the orientation of the extraction site 140. When the first image P1 is a distance image, by changing the gradation, the coordinates in the depth direction of the extraction site 140 change. By changing the position, size, and orientation respectively, when the converted image P3 is superimposed on the second image P2 in the superimposition process 74, the position, size, and orientation of the converted image P3 with respect to the second object 2 change respectively.

[0056] In process 73, the gradation can be changed for each individual divided part 141. Also, when performing a deformation process after the division process, as shown in FIG. 4D, in the deformation process, the rotation center C1 of each of the plurality of divided parts 141 is provided at a position adjacent to another divided part 141. That is, in this case, in the deformation process, at least a part of the divided parts 141 can be rotated about the rotation center C1 to change the orientation of at least a part of the divided parts 141. Also, a part of the divided parts 141 may be moved (on the XY plane) with respect to another divided part 141. The XY plane is orthogonal to the rotation axis (rotation center C1). Also, the size may be changed for each individual divided part 141.

[0057] In the superimposing process 74, a superimposed image P4 in which the converted image P3 is superimposed on the second image P2 is generated. That is, after the converted image P3 as shown in FIG. 4C is generated by the image conversion process, the converted image P3 is superimposed on the second image P2 (see FIG. 3), and as shown in FIG. 5, the superimposed image P4 is generated.

[0058] Here, the converted image P3 is an image obtained by processing the defective part 14 of the first object 1. Therefore, the object 4 shown in the superimposed image P4 includes the defective part 44 generated by processing the defective part 14. That is, the superimposed image P4 is an image (defective product image) in which at least the defective occurrence location E4 of the object 4 is displayed.

[0059] Also, the superimposed image P4 is a three-dimensional image displayed with the depth as gradation. That is, the superimposed image P4 includes information on the coordinates in the depth direction of each of the converted image P3 and the second image P2 as gradation.

[0060] The superimposing process 74 preferably includes an interpolation process for interpolating the boundary between the converted image P3 and the second image P2 so as to smoothly connect the converted image P3 and the second image P2. Thereby, a more natural superimposed image P4 can be generated. The interpolation process is realized, for example, by linear interpolation.

[0061] (6) Setting information Next, the setting information 81 that defines the processing performed by the image processing unit 53 will be described. The setting information 81 is information regarding the process of generating the superimposed image P4 from the first image P1 and the second image P2. More specifically, the setting information 81 includes information regarding at least one of the extraction process 71, the image conversion processes (processes 72 and 73), and the superimposition process 74.

[0062] The setting information input unit 54 acquires the setting information 81. The setting information input unit 54 acquires the setting information 81 from a user interface 55 that receives an input operation of the setting information 81 by the user. The user interface 55 includes at least one of, for example, a mouse, a keyboard, and a touch pad.

[0063] The setting information 81 includes one or more of the following information. The first information that the setting information 81 may include is information for determining the range of the extraction site 140 in the first image P1. The second information that the setting information 81 may include is information for determining the number of divisions of the extraction site 140 in the image conversion process. The third information that the setting information 81 may include is information for determining at least one change among the gradation, position, size, and orientation of the extraction site 140 in the image conversion process. The fourth information that the setting information 81 may include is information for determining the mixing ratio between the converted image P3 and the second image P2 in the superimposition process 74. The fifth information that the setting information 81 may include is information regarding at least one of the gradation, position, size, and orientation (rotation angle) of each of the plurality of divided parts 141. The sixth information that the setting information 81 may include is information on the expansion / contraction rate in the process of expanding or contracting the extraction site 140 in a predetermined direction.

[0064] The image processing unit 53 can, for example, set the range selected by the user using the user interface 55 as the range of the extraction site 140. Further, the image processing unit 53 can, for example, set the number of divisions input by the user using the user interface 55 as the number of divisions of the extraction site 140, and generate the same number of division parts 141 as the number of divisions. Further, the image processing unit 53 can, for example, change the tone, position, size, and orientation of each division part 141 or the entire extraction site 140 according to the information input by the user using the user interface 55. Further, the image processing unit 53 can, for example, superimpose the converted image P3 and the second image P2 at the mixing ratio input by the user using the user interface 55. The mixing ratio is, for example, the α value of each of the converted image P3 and the second image P2. The image processing unit 53 superimposes the converted image P3 on the second image P2 by α blending.

[0065] The setting information generation unit 56 generates setting information 81. The setting information generation unit 56 determines, for example, the range of the extraction site 140 in the first image P1 using a predetermined learned model for detecting the defective site 14 (extraction site 140) from the image. Further, the setting information generation unit 56 determines, for example, the number of divisions of the extraction site 140 according to the size of the extraction site 140.

[0066] Note that the setting information 81 generated by the setting information generation unit 56 may have randomness. For example, the setting information generation unit 56 may randomly determine at least one of the position, size, and orientation of the extraction site 140 (converted image P3) with respect to the second object 2.

[0067] Randomness is not limited to all events occurring with equal probability. The setting information generation unit 56 can, for example, set parameters for randomly determining the position of the extraction site 140 by referring to information on the occurrence probability of defects in each of a plurality of regions on the second object 2, so that regions with a higher occurrence probability are more likely to be selected as the position of the extraction site 140.

[0068] Further, the setting information generation unit 56 may assign a label to the superimposed image P4. The setting information generation unit 56 may determine the label of the superimposed image P4 according to, for example, the label assigned to the first image P1. Specifically, when the label assigned to the first image P1 is the label "defective product" and the extraction site 140 in the first image P1 coincides with the defective occurrence location E1, the setting information generation unit 56 may set the label of the superimposed image P4 to the label "defective product". In this case, the label may include the type of defect, and the type of defect may be the same as the type of defect in the first image P1.

[0069] (7) Display output unit and display device The display output unit 57 outputs information to the display device 58. The display device 58 performs a display according to the information received from the display output unit 57.

[0070] The display device 58 includes a display. The display is, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, etc. The display device 58 displays, for example, the first image P1, the second image P2, the converted image P3, and the superimposed image P4. The display device 58 is used as an output interface that performs a display related to the setting information 81 when the user inputs the setting information 81 to the user interface 55. For example, when the first image P1 is displayed on the display device 58, the user can determine the range of the extraction site 140, etc. by operating the user interface 55 to move the cursor displayed on the display device 58. Further, the user can input the gradation, orientation of the extraction site 140, or the α value of each of the converted image P3 and the second image P2 by, for example, aligning the cursor with the slider displayed on the display device 58 and performing a drag operation.

[0071] In addition, the display output unit 57 outputs information regarding a plurality of divided parts 141 to the display device 58. Based on the information regarding the plurality of divided parts 141 acquired from the display output unit 57, the display device 58 displays the plurality of divided parts 141. As a result, the user can select a desired divided part 141 and input setting information 81 regarding the selected divided part 141.

[0072] In addition, the display device 58 displays a determination result 83 output from a determination unit 61 described later.

[0073] (8) Types of Defects In the present embodiment, the first object 1 is an article formed by welding two or more base materials (here, the first metal plate 11 and the second metal plate 12). That is, the first object 1 is a welded product.

[0074] Examples of types of defects occurring in the welded product include pits, spatter, protrusions, holes, and undercuts. The extraction site 140 includes at least a part of the occurrence location E1 of a defect related to at least one of pits, spatter, protrusions, holes, and undercuts occurring in the welded product. In the present embodiment, the extraction site 140 includes all of the occurrence location E1 of the defect, and the defect is an undercut. The gradation of the extraction site 140 corresponds to the depth of the undercut.

[0075] A pit is a depression occurring in the bead 13. Spatter is a protrusion having a spherical or conical shape or the like occurring in the bead 13. A protrusion is a columnar protrusion occurring in the bead 13. Having a hole means that a part of the bead 13 melts and falls off and is missing. An undercut is a depression occurring around the bead 13.

[0076] The extraction site 140 is not limited to the defective site 14 and may be a site where no defect has occurred. Alternatively, the extraction site 140 may include the defective site 14 and a site in the vicinity of the defective site 14. Further, the extraction site 140 may include a plurality of defective sites 14.

[0077] In particular, when the defect occurring in the first object 1 is a defect occurring in a narrow range (such as a pit, sputtering, or protrusion), it is preferable that the extraction site 140 includes not only one defective site 14 but also a site in the vicinity of the defective site 14 or another defective site 14. Thereby, the length of the extraction site 140 can be ensured, and the process of bending the extraction site 140 or the process of dividing the extraction site 140 can be made more significant processes.

[0078] (9) Machine learning and pass / fail determination The image output unit 59 outputs the superimposed image P4 generated by the image processing unit 53 to the learning unit 60. The learning unit 60 performs machine learning using the superimposed image P4 as learning data (learning data set). Thereby, the learning unit 60 generates a learned model 82. As described above, the learning unit 60 may use not only the superimposed image P4 but also the first image P1 and the second image P2 as learning data.

[0079] The learning data set is generated by assigning a label indicating "good product" or "defective product" to a plurality of image data, and in the case of a defective product, the type of defect and the position of the defect. The work related to the assignment of the label is performed, for example, by the user via the user interface 55 or by the setting information generation unit 56. The learning unit 60 generates a learned model 82 by performing machine learning on the state of the object (good state, defective state, type of defect, position of defect, etc.) using the learning data set.

[0080] The learning unit 60 may improve the performance of the learned model 82 by performing re-learning using a learning data set including newly acquired learning data. For example, if a new type of defect is found in the object, it is also possible to cause the learning unit 60 to perform re-learning regarding the new defect.

[0081] In a factory production line or the like, the imaging device captures an object and generates an inspection image P5. More specifically, the imaging device captures an object on which a bead has been formed after actually finishing the welding process and generates an inspection image P5. The inspection image acquisition unit 62 acquires the inspection image P5 from the imaging device. The determination unit 61 performs a pass / fail determination on the inspection image P5 (object) acquired by the inspection image acquisition unit 62 using the learned model 82 generated by the learning unit 60. Further, when the object is a defective product, the determination unit 61 determines what type of defect it is and where the defect is located. The determination unit 61 outputs a determination result 83. The determination result 83 is output to, for example, the display device 58, and the display device 58 displays the determination result 83. The user can confirm the determination result 83 through the display device 58. Also, for an object determined to be a "defective product" by the determination unit 61, the production equipment may be controlled to discard it before it is conveyed to the next process. Further, the determination result 83 is output to, for example, a data server and stored in the data server.

[0082] As described above, the image generation system 5 includes an inspection image acquisition unit 62 that acquires the inspection image P5, and a determination unit 61 that performs a pass / fail determination on the inspection image P5 using the learned model 82. The learned model 82 is generated based on the superimposed image P4 generated by the image processing unit 53.

[0083] (10) Operation flow Next, with reference to FIG. 6, the flow of the process in which the image generation system 5 generates the superimposed image P4 will be described. Note that the flow shown in FIG. 6 is merely an example, and the order of the processes may be changed as appropriate, or processes may be added or omitted as appropriate.

[0084] The setting information 81 acquired in steps ST2 to ST5 and ST7 may be input by the user via the user interface 55, or may be generated by the setting information generation unit 56. Also, a part of the setting information 81 may be input by the user via the user interface 55, and another part of the setting information 81 may be generated by the setting information generation unit 56.

[0085] First, the first image acquisition unit 51 acquires a first image P1 (step ST1). Next, the image processing unit 53 acquires setting information 81 regarding extraction settings and performs an extraction process 71 based on the extraction settings (step ST2). As a result, an extracted image is generated (see FIG. 4A). The setting information 81 regarding the extraction settings is information for determining the range of the extraction site 140 in the first image P1.

[0086] Next, the image processing unit 53 acquires setting information 81 regarding deformation settings and performs a deformation process based on the deformation settings (step ST3). As a result, a deformed image is generated (see FIG. 4B). The setting information 81 regarding the deformation settings is information referred to for deforming the extraction site 140 to be bent, and includes, for example, information indicating the bending location and the bending angle.

[0087] Next, the image processing unit 53 acquires setting information 81 regarding division settings and performs a division process based on the division settings (step ST4). The setting information 81 regarding the division settings includes, for example, information for determining the number of divisions of the extraction site 140.

[0088] Next, the image processing unit 53 acquires setting information 81 regarding conversion settings and performs a conversion process based on the conversion settings (step ST5). The setting information 81 regarding the conversion settings is information for determining at least one change among the gradation, position, size, and orientation of the extraction site 140. The conversion process is a process of changing at least one of the gradation, position, size, and orientation of the extraction site 140. Also, in the conversion process, the gradation, position, size, and orientation (rotation angle) can be changed for each divided part 141.

[0089] Next, the second image acquisition unit 52 acquires the second image P2 (step ST6). Further, the image processing unit 53 acquires the setting information 81 regarding the superimposition setting, and performs the superimposition process 74 based on the superimposition setting (step ST7). The setting information 81 regarding the superimposition setting includes, for example, information for determining the mixing ratio between the converted image P3 and the second image P2 in the superimposition process 74.

[0090] Thereafter, the image processing unit 53 outputs the superimposed image P4 generated by the superimposition process 74 (step ST8).

[0091] The image processing unit 53 determines whether or not a prescribed number (or a number greater than the prescribed number) of superimposed images P4 have been output (step ST9). When the output of the prescribed number of superimposed images P4 is completed, the image processing unit 53 ends the process of generating the superimposed image P4. On the other hand, when the output of the prescribed number of superimposed images P4 has not been completed, the process returns to step ST1. By repeating steps ST1 to ST8, a prescribed number of superimposed images P4 are generated.

[0092] The minimum required number of the first images P1 is one, and the minimum required number of the second images P2 is one. When generating one superimposed image P4 and when generating another superimposed image P4, at least one of the first image P1 and the second image P2 may be changed.

[0093] Also, a certain superimposed image P4 generated by the image processing unit 53 may be used as the first image P1 or the second image P2 when generating another superimposed image P4.

[0094] (11) Advantages According to the image generation system 5 of this embodiment, a superimposed image P4 suitable for machine learning can be generated. In particular, since the extraction site 140 is deformed along the shape of the second object 2 in the second image P2, the superimposed image P4 can be made into a natural image as compared with the case where the image of the extraction site 140 is superimposed on the second image P2 without such deformation. That is, defects that actually occur in the second object 2 can be reproduced in the superimposed image P4. Therefore, the learned model 82 generated using the superimposed image P4 becomes a model that can accurately determine the presence or absence of defects and the type of defects.

[0095] In addition, the image of the extraction site 140 is processed for each divided part 141. As a result, compared with the case where the extraction site 140 is not divided, it becomes possible to easily determine a parameter (setting information 81) for determining a process of processing the image of the extraction site 140, and it becomes easier to generate a superimposed image P4 desired by a user or the like. Therefore, it is possible to improve the accuracy of machine learning using the superimposed image P4.

[0096] (Modification Example 1) Hereinafter, Modification Example 1 will be described with reference to FIGS. 7A to 7E. Since the configuration of the image generation system 5 is common to the above-described embodiment and this Modification Example 1, the same components as those in the embodiment are denoted by the same reference numerals and the description thereof is omitted.

[0097] Similar to the embodiment, the setting information 81 regarding the process of generating the superimposed image P4 from the first image P1 and the second image P2 may be input by the user via the user interface 55, or may be generated by the setting information generation unit 56. Further, a part of the setting information 81 may be input by the user via the user interface 55, and another part of the setting information 81 may be generated by the setting information generation unit 56.

[0098] In this Modification Example 1, after the division process of dividing the extraction site 140 into a plurality of divided parts 141, the image processing unit 53 performs a process of deforming the extraction site 140 along the shape of the second object 2 in the second image P2.

[0099] First, similar to the embodiment, the image processing unit 53 performs a process of extracting the extraction site 140 from among the first objects 1 shown in the first image P1 (see FIG. 2). As a result, an extraction image in which the extraction site 140 is extracted is generated as shown in FIG. 7A. Further, the image processing unit 53 may appropriately perform a process of inverting (mirror-inverting) the extraction site 140.

[0100] Next, the image processing unit 53 performs a stretching / compressing process of stretching or compressing the extraction site 140 in a predetermined direction. As a result, a transformed image P3 as shown in FIG. 7B is generated. The stretching / compressing process may be omitted as appropriate.

[0101] Next, the image processing unit 53 performs a splitting process. As a result, a transformed image P3 as shown in FIG. 7C is generated.

[0102] Furthermore, the image processing unit 53 processes the image of the extraction site 140 for each individual split part 141. The image processing unit 53 performs at least a process of deforming (hereinafter referred to as a bending process) the extraction site 140 along the shape of the second object 2 in the second image P2.

[0103] The rotation center C1 of each of the plurality of split parts 141 is provided at a position adjacent to the other split parts 141 (see FIG. 7E). In the bending process, the extraction site 140 is deformed by rotating at least some of the split parts 141 about their respective rotation centers C1. As a result of the bending process, a transformed image P3 as shown in FIG. 7D is generated.

[0104] The bending process may include a process of correcting the transformed image P3 after rotating at least some of the split parts 141. For example, the transformed image P3 may be corrected so as to smoothly connect the plurality of split parts 141.

[0105] Furthermore, the image processing unit 53 performs a conversion process (process 73) on at least some of the divided parts 141. The conversion process is a process of changing at least one of the gradation, position, size, and orientation of the extraction part 140. Here, in the conversion process, the gradation, position, size, and orientation can be changed for each divided part 141.

[0106] After the conversion process, the image processing unit 53 superimposes the converted image P3 on the second image P2 to generate a superimposed image P4 as shown in FIG. 5.

[0107] As described above, in this modification example, the extraction part 140 is divided into a plurality of divided parts 141, and the bending process is realized by rotating at least some of the divided parts 141. Therefore, even when the user determines and inputs the setting information 81 related to the bending process, the amount of information required to determine the setting information 81 can be suppressed, and the setting information 81 can be easily determined.

[0108] (Other Modification Examples of the Embodiment) Hereinafter, other modification examples of the embodiment will be listed. The following modification examples may be realized in appropriate combinations. Also, the following modification examples may be realized in appropriate combination with the above-described modification example 1.

[0109] The learned model 82 is not limited to the model used for welding appearance inspection, and may be a model used for various inspections and the like. Also, the learned model 82 is not limited to the model used for inspection, and may be a model used for various image recognitions.

[0110] The main body that executes the extraction process 71 is not limited to the image processing unit 53, and may be a configuration external to the image generation system 5.

[0111] At least one of the user interface 55, the display device 58, the learning unit 60, and the determination unit 61 may be a configuration external to the image generation system 5.

[0112] The display device 58 may be a portable terminal such as a smartphone or a tablet terminal.

[0113] The images (such as the first image P1, the second image P2, the converted image P3, the superimposed image P4, and the inspection image P5, etc.) processed by the image generation system 5 are not limited to three-dimensional images, and may be two-dimensional images or images of four dimensions or more.

[0114] The superimposed image P4 generated from the first image P1 and the second image P2 is not limited to a defective product image, and may be a non-defective product image.

[0115] The first object 1 of the first image P1 may be an article having the same shape as the second object 2 of the second image P2, or may be an article having a different shape.

[0116] The first image P1 may be the same image as the second image P2.

[0117] The first image P1 and the second image P2 may be defective product images or non-defective product images. Alternatively, one of the first image P1 and the second image P2 may be a non-defective product image and the other may be a defective product image.

[0118] The first image P1 may be an image capturing a part of the first object 1. The second image P2 may be an image capturing a part of the second object 2.

[0119] The image processing unit 53 may extract a plurality of extraction parts 140. In this case, a plurality of converted images P3 are generated.

[0120] The image processing unit 53 may generate a plurality of converted images P3 from one extraction part 140.

[0121] In the superimposing process 74, the image processing unit 53 may superimpose a plurality of converted images P3 on the second image P2.

[0122] In the converted image P3, it is not essential that the plurality of divided parts 141 are connected to each other, and a certain divided part 141 may be provided separately from other divided parts 141.

[0123] The rotation center C1 of the divided part 141 may be provided at a position separated from the divided part 141.

[0124] The first image P1, the second image P2, the converted image P3, the superimposed image P4, and the inspection image P5 may be luminance image data representing the luminance of the object by gradation.

[0125] In the embodiment, the "gradation" is the density of a single color (for example, black). In contrast, the "gradation" may be the density of each of a plurality of colors (for example, three colors of RGB).

[0126] The range in the second image P2 where the converted image P3 is to be arranged may be set by the user operating the user interface 55. For example, the range in the second image P2 where the converted image P3 is to be arranged may be selectable from among the first metal plate 21, the second metal plate 22, and the bead 23. When the first metal plate 21 is selected as the range in the second image P2 where the converted image P3 is to be arranged, the image processing unit 53 may arrange the converted image P3, for example, at a randomly determined position within the first metal plate 21.

[0127] The execution entity of the image generation system 5 or the image generation method in the present disclosure includes a computer system. The computer system mainly consists of a processor and a memory as hardware. By the processor executing a program recorded in the memory of the computer system, at least a part of the functions as the execution entity of the image generation system 5 or the image generation method in the present disclosure is realized. The program may be pre-recorded in the memory of the computer system, may be provided through a telecommunication line, or may be provided by being recorded on a non-transitory recording medium such as a memory card, an optical disk, or a hard disk drive that can be read by the computer system. The processor of the computer system is composed of one or more electronic circuits including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). Here, integrated circuits such as the IC or LSI are called differently depending on the degree of integration, and include integrated circuits called system LSI, VLSI (Very Large Scale Integration), or ULSI (Ultra Large Scale Integration). Furthermore, for an FPGA (Field-Programmable Gate Array) that is programmed after the manufacture of the LSI, or a logic device capable of reconfiguring the bonding relationship inside the LSI or reconfiguring the circuit section inside the LSI, it can also be adopted as a processor. The one or more electronic circuits may be integrated on one chip, or may be provided distributed on a plurality of chips. The plurality of chips may be integrated in one device, or may be provided distributed in a plurality of devices. The computer system mentioned here includes a microcontroller having one or more processors and one or more memories. Therefore, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

[0128] Also, it is not an essential configuration of the image generation system 5 that a plurality of functions in the image generation system 5 are aggregated in one device, and the components of the image generation system 5 may be provided dispersedly in a plurality of devices. Further, at least some functions of the image generation system 5, for example, at least one of the image processing unit 53, the learning unit 60, and the determination unit 61, may be realized by a server or a cloud (cloud computing) or the like. Conversely, a plurality of functions of the image generation system 5 may be aggregated in one device.

[0129] (Summary) From the embodiments and the like described above, the following aspects are disclosed.

[0130] The image generation system (5) according to the first aspect includes a first image acquisition unit (51), a second image acquisition unit (52), and an image processing unit (53). The first image acquisition unit (51) acquires a first image (P1) obtained by imaging a first object (1). The second image acquisition unit (52) acquires a second image (P2) obtained by imaging a second object (2). The image processing unit (53) performs an image conversion process and a superimposition process (74). The image conversion process is a process of generating a converted image (P3) which is an image obtained by processing a predetermined extraction part (140) of the first object (1) based on the first image (P1). The superimposition process (74) is a process of generating a superimposed image (P4) by superimposing the converted image (P3) on the second image (P2). The image conversion process includes a process of dividing the extraction part (140) into a plurality of divided parts (141), and a process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts (141).

[0131] According to the above configuration, an image suitable for machine learning (superimposed image (P4)) can be generated. In particular, before generating the superimposed image (P4) by superimposing the converted image (P3) on the second image (P2), a process of dividing the extraction part (140) into a plurality of divided parts (141) is performed, and further, a process (conversion process) of changing at least one of the gradation, position, size, and orientation of each of at least one divided part (141) is performed. That is, the conversion process is performed for each divided part (141). As a result, compared with the case where the extraction part (140) is not divided, it becomes possible to easily determine the parameters (setting information (81)) for determining the conversion process, and it becomes easy to generate the superimposed image (P4) desired by the user or the like. Therefore, it is possible to improve the accuracy of machine learning using the superimposed image (P4).

[0132] Further, the image generation system (5) according to the second aspect further includes a setting information input unit (54) in the first aspect. The setting information input unit (54) acquires setting information (81) regarding the process of generating the superimposed image (P4) from the first image (P1) and the second image (P2). The setting information input unit (54) acquires the setting information (81) from a user interface (55) that receives an input operation of the setting information (81) by the user.

[0133] According to the above configuration, the user can specify the setting information (81).

[0134] Further, the image generation system (5) according to the third aspect further includes a setting information generation unit (56) in the first or second aspect. The setting information generation unit (56) generates setting information (81) regarding the process of generating the superimposed image (P4) from the first image (P1) and the second image (P2).

[0135] According to the above configuration, the setting information (81) is generated even if the user does not specify the setting information (81).

[0136] In addition, in the image generation system (5) according to the fourth aspect, in the second or third aspect, the setting information (81) includes one or more of the following first to fourth pieces of information. The first piece of information that the setting information (81) may include is information for determining the range of the extraction site (140) in the first image (P1). The second piece of information that the setting information (81) may include is information for determining the number of divisions of the extraction site (140) in the image conversion process. The third piece of information that the setting information (81) may include is information for determining at least one change among the gradation, position, size, and orientation of at least one divided part (141) in the image conversion process. The fourth piece of information that the setting information (81) may include is information for determining the mixing ratio between the converted image (P3) and the second image (P2) in the superimposition process (74).

[0137] According to the above configuration, the converted image (P3) can be generated according to the content of the setting information (81).

[0138] In addition, in the image generation system (5) according to the fifth aspect, in any one of the first to fourth aspects, the first image (P1) is an image obtained by imaging the defective occurrence location (E1) of the first object (1). The second image (P2) is an image obtained by imaging the second object (2) as a non-defective product. The extraction site (140) includes at least a part of the defective occurrence location (E1).

[0139] According to the above configuration, a defective product image can be generated as the superimposed image (P4).

[0140] In addition, in the image generation system (5) according to the sixth aspect, in the fifth aspect, the first object (1) is a welded object. The extraction site (140) includes at least a part of the defective occurrence location (E1) related to at least one of pits, spatter, protrusions, perforations, and undercuts occurring in the welded object.

[0141] According to the above configuration, a defective product image of the welded object can be generated as the superimposed image (P4).

[0142] Also, the image generation system (5) according to the seventh aspect further includes an image output unit (59) in any one of the first to sixth aspects. The image output unit (59) outputs the superimposed image (P4) generated by the image processing unit (53) to the learning unit (60). The learning unit (60) performs machine learning using the superimposed image (P4) as learning data.

[0143] According to the above configuration, a highly accurate learned model (82) can be generated by machine learning based on the superimposed image (P4).

[0144] Also, the image generation system (5) according to the eighth aspect further includes an inspection image acquisition unit (62) and a determination unit (61) in any one of the first to seventh aspects. The inspection image acquisition unit (62) acquires an inspection image (P5). The determination unit (61) uses the learned model (82) to determine whether the inspection image (P5) is good or bad. The learned model (82) is generated based on the superimposed image (P4) generated by the image processing unit (53).

[0145] According to the above configuration, by using the learned model (82) generated based on the superimposed image (P4), a highly accurate pass / fail determination can be made.

[0146] Also, in the image generation system (5) according to the ninth aspect, in any one of the first to eighth aspects, the superimposed image (P4) is a three-dimensional image in which the depth is displayed as a gradation.

[0147] According to the above configuration, machine learning considering depth information can be performed.

[0148] Also, the image generation system (5) according to the tenth aspect further includes a display output unit (57) in any one of the first to ninth aspects. The display output unit (57) outputs information regarding a plurality of divided parts (141) to the display device (58). The display device (58) displays the plurality of divided parts (141) based on the information regarding the plurality of divided parts (141) acquired from the display output unit (57).

[0149] According to the above configuration, the user can grasp a plurality of divided parts (141).

[0150] Also, in the image generation system (5) according to the eleventh aspect, in any one of the first to tenth aspects, in the image conversion process, each of the plurality of divided parts (141) can be rotated with respect to other divided parts (141).

[0151] According to the above configuration, the divided part (141) can be rotated to bend the image of the extraction site (140).

[0152] Also, in the image generation system (5) according to the twelfth aspect, in the eleventh aspect, in the image conversion process, the rotation center (C1) of each of the plurality of divided parts (141) is provided at a position adjacent to other divided parts (141).

[0153] According to the above configuration, the divided part (141) can be rotationally deformed while maintaining the state where two adjacent divided parts (141) are connected to each other.

[0154] Regarding the configuration other than the first aspect, it is not an essential configuration of the image generation system (5) and can be omitted as appropriate.

[0155] In addition, the image generation method according to the 13th aspect includes a first image acquisition process, a second image acquisition process, an image conversion process, and a superimposition process (74). In the first image acquisition process, a first image (P1) obtained by imaging a first object (1) is acquired. In the second image acquisition process, a second image (P2) obtained by imaging a second object (2) is acquired. The image conversion process is a process of generating a converted image (P3) that is an image obtained by processing a predetermined extraction part (140) of the first object (1) based on the first image (P1). The superimposition process (74) is a process of generating a superimposed image (P4) by superimposing the converted image (P3) on the second image (P2). The image conversion process includes a process of dividing the extraction part (140) into a plurality of divided parts (141) and a process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts (141).

[0156] According to the above configuration, an image (superimposed image (P4)) suitable for machine learning can be generated.

[0157] In addition, the image generation method according to the 14th aspect further includes a setting information generation process of generating setting information (81) related to the process of generating the superimposed image (P4) from the first image (P1) and the second image (P2) in the 13th aspect.

[0158] According to the above configuration, the setting information (81) is generated without the user specifying the setting information (81).

[0159] Regarding the configuration other than the 13th aspect, it is not an essential configuration of the image generation method and can be omitted as appropriate.

[0160] In addition, the program according to the 15th aspect is a program for causing one or more processors of a computer system to execute the image generation method according to the 13th or 14th aspect.

[0161] According to the above configuration, an image (superimposed image (P4)) suitable for machine learning can be generated.

[0162] Not limited to the above aspects, various configurations (including modifications) of the image generation system (5) according to the embodiment can be embodied in an image generation method, a (computer) program, or a non-transitory recording medium storing the program.

Explanation of Signs

[0163] 1 First object 2 Second object 5 Image generation system 51 First image acquisition unit 52 Second image acquisition unit 53 Image processing unit 54 Setting information input unit 55 User interface 56 Setting information generation unit 57 Display output unit 58 Display device 59 Image output unit 60 Learning unit 61 Judgment unit 62 Inspection image acquisition unit 74 Superposition processing 81 Setting information 82 Learned model 140 Extraction site 141 Division part C1 Rotation center E1 Location of defect occurrence P1 First image P2 Second image P3 Transformed image P4 Superposed image P5 Inspection image

Claims

1. A first image acquisition unit that acquires a first image obtained by imaging a first object; A second image acquisition unit that acquires a second image obtained by imaging a second object; An image processing unit that performs an image conversion process for generating a converted image that is an image obtained by processing a predetermined extraction part of the first object based on the first image, and a superimposition process for generating a superimposed image obtained by superimposing the converted image on the second image; The image conversion process includes: A process of dividing the extraction part into a plurality of divided parts; A process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts; An image generation system.

2. The image generation system further includes a setting information input unit that acquires setting information regarding a process of generating the superimposed image from the first image and the second image, The setting information input unit acquires the setting information from a user interface that receives an input operation of the setting information by a user. The image generation system according to claim 1.

3. The image generation system further includes a setting information generation unit that generates setting information regarding a process of generating the superimposed image from the first image and the second image. The image generation system according to claim 1 or 2.

4. The setting information includes Information for determining the range of the extraction part in the first image; Information for determining the number of divisions of the extraction part in the image conversion process; Information for determining at least one change among the gradation, position, size, and orientation of the at least one divided part in the image conversion process; Information for determining a mixing ratio between the converted image and the second image in the superimposition process, and includes one or more of them. The image generation system according to claim 2.

5. The first image is an image obtained by imaging a defective occurrence location of the first object, The second image is an image obtained by imaging the second object as a non-defective product, The extraction part includes at least a part of at least one defective occurrence location of the defective occurrence location. The image generation system according to claim 1 or 2.

6. The first object is a welded object, The extraction part includes at least a part of at least one defective occurrence location related to at least one of pits, spatter, protrusions, perforations, and undercuts occurring in the welded object. The image generation system according to claim 5.

7. An image output unit that outputs the superimposed image generated by the image processing unit to a learning unit that performs machine learning using the superimposed image as learning data is further provided. The image generation system according to claim 1 or 2.

8. An inspection image acquisition unit that acquires an inspection image, A determination unit that determines whether the inspection image is good or bad using a learned model generated based on the superimposed image generated by the image processing unit is further provided. The image generation system according to claim 1 or 2.

9. The superimposed image is a three-dimensional image in which depth is displayed as gradation. The image generation system according to claim 1 or 2.

10. A display output unit that outputs information about the plurality of divided parts to a display device is further provided. The display device displays the plurality of divided parts based on the information about the plurality of divided parts acquired from the display output unit. The image generation system according to claim 1 or 2.

11. In the image conversion process, each of the plurality of divided parts is rotatable with respect to other divided parts. The image generation system according to claim 1 or 2.

12. In the image conversion process, the rotation center of each of the plurality of divided parts is provided at a position adjacent to other divided parts. The image generation system according to claim 11.

13. A first image acquisition process for acquiring a first image obtained by imaging a first object, A second image acquisition process for acquiring a second image obtained by imaging a second object, An image conversion process for generating a converted image that is an image obtained by processing a predetermined extraction site of the first object based on the first image, A superimposing process for generating a superimposed image in which the converted image is superimposed on the second image, and The image conversion process includes A process of dividing the extraction site into a plurality of divided parts, A process of changing at least one of the gradation, position, size, and orientation of at least one of the plurality of divided parts. Image generation method.

14. The method further includes a setting information generation process for generating setting information regarding a process of generating the superimposed image from the first image and the second image. The image generation method according to claim 13.

15. For causing one or more processors of a computer system to execute the image generation method according to claim 13 or 14, Program.

Images