Image generation system, image generation method, and program

Abstract

The purpose of the present disclosure is to generate an image suited for machine learning. An image generation system (5) comprises a first image acquisition unit (51), a second image acquisition unit (52), and an image processing unit (53). The image processing unit (53) performs image transformation processing (processing 72, 73) and superimposing processing (74). The image transformation processing is processing for generating, on the basis of a first image (P1), a transformation image (P3) which results from processing a prescribed extracted section of a first object. The superimposing processing (74) is processing for generating a superimposed image (P4) in which the transformation image (P3) is superimposed on a second image (P2). The image transformation processing includes processing for transforming the extracted section in accordance with the shape of a second object in the second image (P2).

Description

[Technical Field]

[0001] The present disclosure generally relates to an image generation system, an image generation method, and a program, and more particularly to an image generation system, an image generation method, and a program that generate an image by superimposing multiple images. [Background technology]

[0002] The image display device (image generation system) described in Patent Document 1 includes an image designation means, an image processing means, an image synthesis means, and an image display means. The image designation means designates an original image to be processed. The image processing means processes the original image to generate a processed image. The image synthesis means replaces a partial area of ​​the image with a corresponding partial area of ​​the processed image to generate an entire image. The image display means displays the entire entire image or a partial image including the boundary.

[0003] In this way, the image display device described in Patent Document 1 can generate a new image from an original image. However, it is difficult for this image display device to generate an image suitable for machine learning. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-3603 Summary of the Invention

[0005] The present disclosure aims to provide an image generation system, an image generation method, and a program that can generate images suitable for machine learning.

[0006] An image generation system according to one aspect of the present disclosure includes a first image acquisition unit, a second image acquisition unit, and an image processing unit. The first image acquisition unit acquires a first image obtained by capturing an image of a first object. The second image acquisition unit acquires a second image obtained by capturing an image of a second object. The image processing unit performs image conversion processing and superimposition processing. The image conversion processing is processing for generating a converted image, which is an image obtained by processing a predetermined extracted portion of the first object based on the first image. The superimposition processing is processing for generating a superimposed image by superimposing the converted image on the second image. The image conversion processing is processing for deforming the extracted portion to conform to the shape of the second object in the second image. and a process of dividing the extracted portion into a plurality of divided parts. Includes: An image generation system according to another aspect of the present disclosure includes a first image acquisition unit, a second image acquisition unit, and an image processing unit. The first image acquisition unit acquires a first image obtained by capturing an image of a first object. The second image acquisition unit acquires a second image obtained by capturing an image of a second object. The image processing unit performs an image conversion process and a superimposition process. The image conversion process is a process of generating a converted image, which is an image obtained by processing a predetermined extracted portion of the first object based on the first image. The superimposition process is a process of generating a superimposed image by superimposing the converted image on the second image. The image conversion process includes a process of deforming the extracted portion to conform to a shape of the second object in the second image. In the image conversion process, deforming the extracted portion to conform to the shape of the second object in the second image means deforming the extracted portion to conform to at least a partial outline or edge of the second object in the second image.

[0007] An image generating method according to one aspect of the present disclosure includes a first image acquisition process, a second image acquisition process, an image conversion process, and a superimposition process. The first image acquisition process acquires a first image obtained by capturing an image of a first object. The second image acquisition process acquires a second image obtained by capturing an image of a second object. The image conversion process is a process of generating a converted image, which is an image obtained by processing a predetermined extracted portion of the first object based on the first image. The superimposition process is a process of generating a superimposed image by superimposing the converted image on the second image. The image conversion process is a process of deforming the extracted portion to conform to the shape of the second object in the second image. and a process of dividing the extracted portion into a plurality of divided parts. Includes: An image generating method according to another aspect of the present disclosure includes a first image acquisition process, a second image acquisition process, an image conversion process, and a superimposition process. The first image acquisition process acquires a first image obtained by capturing an image of a first object. The second image acquisition process acquires a second image obtained by capturing an image of a second object. The image conversion process generates a converted image, which is an image obtained by processing a predetermined extracted portion of the first object based on the first image. The superimposition process generates a superimposed image by superimposing the converted image on the second image. The image conversion process includes a process of deforming the extracted portion to conform to the shape of the second object in the second image. In the image conversion process, deforming the extracted portion to conform to the shape of the second object in the second image means deforming the extracted portion to conform to at least a partial outline or edge of the second object in the second image.

[0008] A program according to one embodiment of the present disclosure includes: Any of the above A program for causing one or more processors of a computer system to execute the image generation method. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram of an image generation system according to an embodiment. [Figure 2]FIG. 2 is a schematic diagram showing an example of a first image used in the image generating system. [Figure 3] FIG. 3 is a schematic diagram showing an example of a second image used in the image generating system. [Figure 4] 4A to 4D are schematic diagrams showing examples of images generated by the image generation system of the same. [Figure 5] FIG. 5 is a schematic diagram showing an example of a superimposed image generated by the image generating system of the same. [Figure 6] FIG. 6 is a flowchart showing the processing of the image generating system. [Figure 7] 7A to 7E are schematic diagrams showing an example of an image generated by an image generation system according to the first modification. DETAILED DESCRIPTION OF THE INVENTION

[0010] (Embodiment) An image generation system 5 according to an embodiment will be described below with reference to the drawings. However, the embodiment described below is merely one of various embodiments of the present disclosure. The embodiment described below can be modified in various ways depending on the design, etc., as long as the object of the present disclosure can be achieved. Furthermore, each diagram described in the embodiment described below is a schematic diagram, and the ratios of the sizes and thicknesses of the components in the diagram do not necessarily reflect the actual dimensional ratios.

[0011] (overview) As shown in Fig. 1, the image generation system 5 according to this embodiment creates a superimposed image P4 from a first image P1 and a second image P2. The superimposed image P4 is used as training data for generating a trained model 82 for an object (e.g., an equivalent of the second object 2 (see Fig. 3)). In other words, the superimposed image P4 is training data used to generate a model by machine learning.

[0012] In the present disclosure, a "model" is a program that, when input information about a recognition target (object) is input, recognizes the state of the recognition target and outputs the recognition result. A "trained model" refers to a model for which machine learning using training data has been completed. Furthermore, "training data (training dataset)" is a dataset that combines input information (images) input to the model with labels assigned to the input information, and is so-called teacher data. In other words, in this embodiment, the trained model 82 is a model for which machine learning using supervised learning has been completed.

[0013] The trained model 82 here may include, for example, a model using a neural network or a model generated by deep learning using a multilayer neural network. The neural network may include, for example, a convolutional neural network (CNN) or a Bayesian neural network (BNN). The trained model 82 is realized by implementing a trained neural network in an integrated circuit such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). The trained model 82 is not limited to a model generated by deep learning. The trained model 82 may also be a model generated by a support vector machine, a decision tree, or the like.

[0014] In this embodiment, as an example, the object to be recognized is a welded work. Fig. 5 is an example of a superimposed image P4 (learning data), and the superimposed image P4 shows an object 4. Like the recognition target, the object 4 is also a welded work. The object 4 includes a first metal plate 41, a second metal plate 42, and a bead 43.

[0015] The bead 43 is formed at the boundary B1 (weld location) between the first metal plate 41 and the second metal plate 42 when two or more base materials (here, the first metal plate 41 and the second metal plate 42) are welded using a metal welding material. The size and shape of the bead 43 mainly depend on the welding material. Furthermore, the object 4 includes a defective portion 44. Hereinafter, the location where the defective portion 44 exists will be referred to as the defect occurrence location E4. In FIG. 5, the defect occurrence location E4 is included in the second metal plate 42.

[0016] When an image (inspection image P5) of a recognition target (object) is input, the trained model 82 recognizes the state of the object and outputs a recognition result. Specifically, the trained model 82 outputs, as a recognition result, whether the object is a good or defective product, and if it is a defective product, outputs the type of defect. In other words, the trained model 82 is used for determining whether an object is good or bad, in other words, for welding appearance inspection to inspect whether welding has been performed correctly.

[0017] Whether an object is a non-defective product is determined based on, for example, whether the bead length, bead height, bead rise angle, bead throat thickness, bead overfill, and misalignment of the bead weld point (including misalignment of the bead start point) are within the allowable range. For example, if any one of the conditions listed above is not within the allowable range, the object is determined to be defective. Also, whether an object is a non-defective product is determined based on, for example, the presence or absence of defective parts such as undercuts on the object, bead pits, bead spatter, and bead protrusions. For example, if any one of the defective parts listed above occurs, the object is determined to be defective.

[0018] In order to perform machine learning of a model, it is necessary to prepare a large number of image data, including defective products, as training data. However, if the frequency of defective products on an object production line is low, the training data required to generate a trained model 82 with a high recognition rate is likely to be insufficient. Therefore, it is conceivable to perform data augmentation on training data (first image P1 and second image P2) obtained by actually capturing images of the object, thereby increasing the amount of training data and performing machine learning of the model. Data augmentation refers to a process of increasing the amount of training data by performing processes such as synthesis, translation, enlargement / reduction, rotation, inversion, or noise addition on the training data. In this embodiment, the data augmentation process generates at least one (or, in the inventor's estimation, many) superimposed image P4, and each of the at least one superimposed image P4 is used as training data. In this embodiment, the superimposed image P4 is generated by performing a process of superimposing (combining) multiple original images on multiple original images (multiple training data) before they are processed by the image generation system 5.

[0019] It is not necessary that the multiple original images (first image P1 and second image P2) used to generate the superimposed image P4 be used as training data for generating the trained model 82. In other words, the training data for generating the trained model 82 may include only the multiple superimposed images P4, or may include, in addition to at least one superimposed image P4, the first image P1 and the second image P2, etc., that are not images generated by the image generation system 5. In other words, the training data for generating the trained model 82 may or may not include the original image before being processed by the image generation system 5. Furthermore, the training data for generating the trained model 82 may include images generated by a system other than the image generation system 5.

[0020] As shown in FIG. 1, the image generation system 5 of this embodiment includes a first image acquisition unit 51, a second image acquisition unit 52, and an image processing unit 53. The first image acquisition unit 51 acquires a first image P1 obtained by capturing an image of a first object 1. The second image acquisition unit 52 acquires a second image P2 obtained by capturing an image of a second object 2. The image processing unit 53 performs image conversion processing (processes 72 and 73) and a superimposition processing 74. The image conversion processing is processing for generating a converted image P3, which is an image obtained by processing a predetermined extracted portion 140 of the first object 1 based on the first image P1. The superimposition processing 74 is processing for generating a superimposed image P4 by superimposing the converted image P3 on the second image P2. The image conversion processing includes processing for deforming the extracted portion 140 to conform to the shape of the second object 2 in the second image P2.

[0021] According to this embodiment, an image (superimposed image P4) suitable for machine learning can be generated. In particular, before generating the superimposed image P4 by superimposing the converted image P3 on the second image P2, a process is performed to deform the extracted portion 140 according to the shape of the second object 2 in the second image P2, and the converted image P3 is generated. As a result, compared to when such a process is not performed, the superimposed image P4 is more likely to resemble an actual image rather than an image generated by superimposition. For example, if the superimposed image P4 is an image of a defective product, the superimposed image P4 is more likely to resemble an image obtained by capturing an image of the actual defective product. Therefore, the accuracy of machine learning using the superimposed image P4 can be improved.

[0022] Furthermore, functions similar to those of the image generation system 5 can be realized by an image generation method. The image generation method of this embodiment includes a first image acquisition process, a second image acquisition process, an image conversion process (processes 72 and 73), and a superimposition process 74. The first image acquisition process acquires a first image P1 obtained by capturing an image of a first object 1. The second image acquisition process acquires a second image P2 obtained by capturing an image of a second object 2. The image conversion process is a process for generating a converted image P3, which is an image obtained by processing a predetermined extracted portion 140 of the first object 1 based on the first image P1. The superimposition process 74 is a process for generating a superimposed image P4 by superimposing the converted image P3 on the second image P2. The image conversion process includes a process for deforming the extracted portion 140 to conform to the shape of the second object 2 in the second image P2.

[0023] Preferably, the image generating method further includes a setting information generating process for generating setting information 81 (see FIG. 1) related to the process of generating a superimposed image P4 from the first image P1 and the second image P2.

[0024] Furthermore, the image generation method is used on a computer system (image generation system 5). In other words, the image generation method can be embodied as a program. The program according to this embodiment is a program for causing one or more processors of the computer system to execute the image generation method according to this embodiment. The program may be recorded on a non-transitory recording medium readable by the computer system.

[0025] (detail) The image generation system 5 according to this embodiment will be described in more detail below.

[0026] (1) Overall structure 1 includes a computer system having one or more processors and a memory. At least some of the functions of the image generation system 5 are realized by the processor of the computer system executing a program stored in the memory of the computer system. The program may be stored in the memory, or may be provided via a telecommunications line such as the Internet, or may be provided by being recorded on a non-transitory recording medium such as a memory card.

[0027] The image generation system 5 may be installed within a factory where the welding will be performed, or at least a portion of the components of the image generation system 5 may be installed outside the factory (for example, in a business establishment located in a different location from the factory).

[0028] As described above, the image generation system 5 has a function of performing data augmentation processing on original images (learning data) to increase the number of pieces of learning data. Hereinafter, a person who uses the image generation system 5 may be simply referred to as a "user." A user is, for example, an operator who monitors a manufacturing process such as a welding process in a factory, or a manager.

[0029] In this embodiment, as described above, the trained model 82 is generated by machine learning. The trained model 82 may be implemented as any type of artificial intelligence or system. Here, the machine learning algorithm is, for example, a neural network. However, the machine learning algorithm is not limited to a neural network and may be, for example, eXtreme Gradient Boosting (XGB) regression, Random Forest, decision tree, logistic regression, support vector machine (SVM), Naive Bayes classifier, k-nearest neighbors, or the like. Furthermore, the machine learning algorithm may be, for example, a Gaussian Mixture Model (GMM), k-means clustering, or the like.

[0030] Furthermore, trained model 82 is not limited to machine learning that classifies inspection images into two classes, good and bad, but may be generated by machine learning that classifies inspection images into multiple classes, such as good, pits, spatter, protrusions, burn-through, and undercut. Pits, spatter, protrusions, burn-through, and undercut are each types of defects. Trained model 82 may also be generated by machine learning that performs object detection and segmentation to detect the positions and types of defective parts in inspection images.

[0031] In addition, in the embodiment, the learning method is, for example, supervised learning, but the learning method is not limited to supervised learning and may be unsupervised learning or reinforcement learning.

[0032] 1, the image generation system 5 includes a storage unit 63 for storing (memorizing) learning data. The storage unit 63 includes a rewritable nonvolatile memory such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). The storage unit 63 may be a memory built into the image generation system 5. However, the storage unit 63 may also be provided outside the image generation system 5.

[0033] 1, the image generation system 5 includes a first image acquisition unit 51, a second image acquisition unit 52, an image processing unit 53, a setting information input unit 54, a user interface 55, a setting information generation unit 56, a display output unit 57, and a display device 58. The image generation system 5 further includes an image output unit 59, a learning unit 60, a determination unit 61, an inspection image acquisition unit 62, and a storage unit 63.

[0034] At least the user interface 55, the display device 58, and the storage unit 63 are tangible components. On the other hand, the first image acquisition unit 51, the second image acquisition unit 52, the image processing unit 53, the setting information input unit 54, the setting information generation unit 56, the display output unit 57, the image output unit 59, the learning unit 60, the determination unit 61, and the inspection image acquisition unit 62 merely indicate functions realized by one or more processors of the image generation system 5, and do not necessarily indicate tangible components.

[0035] (2) First image acquisition unit The first image acquisition unit 51 acquires a first image P1. As shown in FIG. 2, the first image P1 is an image captured of a first object 1. The first image acquisition unit 51 may acquire the first image P1 from a device external to the image generation system 5, or may acquire the first image P1 from the storage unit 63 of the image generation system 5. In the example shown in FIG. 1, the first image acquisition unit 51 acquires the first image P1 from a device external to the image generation system 5. For example, the first image acquisition unit 51 acquires the first image P1 from a computer server.

[0036] Furthermore, the imaging device that captures the image of the first object 1 and generates the first image P1 may be provided in the image generation system 5 or may be provided outside the image generation system 5.

[0037] The first object 1 is a defective product. A defective product is an item in which a defect has occurred. The state of an item that is considered to be in a defective state can be determined appropriately by the user of the image generation system 5.

[0038] The first image P1 is an image (defective product image) of at least the defect location E1 of the first object 1. However, the area captured in the first image P1 is not limited to only the defect location E1, and may also include areas of the first object 1 other than the defect location E1. In the example shown in FIG. 2, the area captured in the first image P1 is the entire first object 1.

[0039] The first object 1 includes a first metal plate 11, a second metal plate 12, and a bead 13. The configurations of the first metal plate 11, the second metal plate 12, and the bead 13 are similar to those of the first metal plate 41, the second metal plate 42, and the bead 43 of the object 4 described above, and therefore a description thereof will be omitted. However, in the first object 1, a defective portion 14 having a shape different from that of the defective portion 44 is present in a location different from the location where the defective portion 44 is provided in the object 4 (defect occurrence location E4). More specifically, the defective portion 14 is present in the first metal plate 11.

[0040] (3) Second image acquisition unit The second image acquisition unit 52 acquires the second image P2. As shown in FIG. 3, the second image P2 is an image captured of the second object 2. The second image acquisition unit 52 may acquire the second image P2 from a device external to the image generation system 5, or may acquire the second image P2 from the storage unit 63 of the image generation system 5. In the example shown in FIG. 1, the second image acquisition unit 52 acquires the second image P2 from a device external to the image generation system 5. For example, the second image acquisition unit 52 acquires the second image P2 from a computer server.

[0041] Furthermore, the imaging device that captures the image of the second object 2 and generates the second image P2 may be provided in the image generation system 5 or may be provided outside the image generation system 5.

[0042] The second object 2 is a non-defective product. A non-defective product is an item that has no defects. 3 In the example shown in FIG. 1, the area captured in the second image P2 is the entire second object 2. However, the area captured in the second image P2 may be only a part of the second object 2.

[0043] The second object 2 includes a first metal plate 21, a second metal plate 22, and a bead 23. The configurations of the first metal plate 21, the second metal plate 22, and the bead 23 are similar to the first metal plate 41, the second metal plate 42, and the bead 43 of the object 4 described above, and therefore a description thereof will be omitted. However, the second object 2 does not have a configuration equivalent to the defective portion 44.

[0044] (4) First image and second image The first image P1 is, for example, a distance image including coordinate information in the depth direction (the direction from the imaging device toward the first object 1). The second image P2 is, for example, a distance image including coordinate information in the depth direction (the direction from the imaging device toward the second object 2). The coordinate information in the depth direction is represented, for example, by gradation. Specifically, the greater the density of a target point in the distance image, the further back the target point is. However, conversely, the lower the density of a target point in the distance image, the further back the target point is may also be represented.

[0045] The imaging device for generating the distance image is a distance image sensor such as a line sensor camera. The imaging device sequentially captures images of multiple objects to generate multiple images. A first image P1 and a second image P2 are selected from the multiple images generated by the imaging device, for example, in response to a user instruction. The image generation system 5 preferably includes an operation unit that accepts instructions regarding the selection. For example, a user interface 55 may be used as the operation unit.

[0046] (5) Image processing unit The image processing unit 53 is realized by, for example, a DSP (Digital Signal Processor) or an FPGA (Field-Programmable Gate Array). The image processing unit 53 performs an extraction process 71, an image conversion process (processes 72 and 73), and a superimposition process 74. The image processing unit 53 performs the extraction process 71, the image conversion process (processes 72 and 73), and the superimposition process 74 based on setting information 81. As will be described later, the setting information 81 may be input by a user or may be automatically generated by the setting information generation unit 56.

[0047] In this embodiment, an image generated by the image conversion process (processes 72 and 73) is called a converted image P3. An image generated during the image conversion process (processes 72 and 73) is also called a converted image P3. Furthermore, an image generated by the superimposition process 74 is called a superimposed image P4.

[0048] The extracted portion 140 is a part of the first object 1. The extraction process 71 is a process for extracting the extracted portion 140 from the first object 1 that appears in the first image P1 (see FIG. 2). That is, an extracted image is generated from the first image P1 by extracting the extracted portion 140, as shown in FIG. 4A.

[0049] In this embodiment, the extracted portion 140 coincides with the defective portion 14. The defective portion 14 coincides with the location E1 where the defect occurred. In other words, the first image P1 is an image of the location E1 where the defect occurred in the first object 1, the second image P2 is an image of the second object 2 as a non-defective product, and the extracted portion 140 includes at least a part (in this embodiment, the entirety) of the location E1 where the defect occurred.

[0050] The process 72 includes a transformation process and a division process. Either the transformation process or the division process may be performed first. In this embodiment, the description will be given assuming that at least a part of the transformation process is performed first, and then the division process is performed.

[0051] The deformation process is a process of deforming the extracted portion 140 so as to bend it. More specifically, the deformation process is a process of deforming the extracted portion 140 so that it conforms to the shape of the second object 2 in the second image P2. The deformation process may also include a process of stretching or compressing the extracted portion 140 in a predetermined direction. The deformation process may also include a process of inverting (mirroring) the extracted portion 140. By performing the deformation process on the extracted image, for example, a converted image P3 such as that shown in FIG. 4B is generated. Then, in the superposition process 74, the extracted portion 140 is arranged so as to conform to the shape of the second object 2.

[0052] By deforming the extracted portion 140 to conform to the shape of the second object 2, for example, a linear extracted portion 140 may become curved, or conversely, a curved extracted portion 140 may become linear. Alternatively, a curved extracted portion 140 may become a curved extracted portion 140 with a different shape from the extracted portion 140 before deformation. Deformations other than those listed here may also be made to the extracted portion 140.

[0053] The process 72 may include a process of extracting a contour or edge of at least a portion of the second object 2 in the second image P2. For example, a portion in the second image P2 where the brightness changes more than a threshold value may be extracted as the contour or edge. Then, in the deformation process, the image processing unit 53 may deform the extracted portion 140 so as to bend it along the contour or edge. That is, in the image conversion process, deforming the extracted portion 140 to conform to the shape of the second object 2 in the second image P2 may mean deforming the extracted portion 140 along at least a portion of the contour or edge of the second object 2 in the second image P2. In this case, in the superimposition process 74, the extracted portion 140 is arranged along the at least a portion of the contour or edge of the second object 2. In FIG. 4B, the extracted portion 140 is deformed along the contour or edge of the upper right to lower right portion of the contour of the bead 43 of the second object 2 shown in FIG. 5.

[0054] The division process is a process of dividing the extracted portion 140 into a plurality of divided parts 141. After the transformation process is performed on the extracted image, the division process is performed to generate, for example, a converted image P3 as shown in FIG. 4C. In this way, the image transformation process includes a process of dividing the extracted portion 140 into a plurality of divided parts 141. In FIG. 4C, the boundaries of each divided part 141 are represented by rectangular frames. FIG. 4D is an enlarged view of a portion of FIG. 4C.

[0055] Process 73 includes a process of changing at least one of the gradation of the extracted portion 140, the position of the extracted portion 140, the size of the extracted portion 140, and the orientation of the extracted portion 140. In other words, the image conversion process includes a process of changing at least one of the gradation of the extracted portion 140, the position of the extracted portion 140, the size of the extracted portion 140, and the orientation of the extracted portion 140. If the first image P1 is a distance image, changing the gradation changes the depth coordinate of the extracted portion 140. Changing the position, size, and orientation changes the position, size, and orientation of the converted image P3 relative to the second object 2 when the converted image P3 is superimposed on the second image P2 in the superimposition process 74.

[0056] In process 73, the gradation can be changed for each divided part 141. Furthermore, when a transformation process is performed after the division process, as shown in FIG. 4D, in the transformation process, the rotation center C1 of each of the multiple divided parts 141 is set at a position adjacent to the other divided parts 141. That is, in this case, in the transformation process, at least some of the divided parts 141 can be rotated around the rotation center C1 to change the orientation of at least some of the divided parts 141. Furthermore, some of the divided parts 141 may be moved (on the XY plane) relative to the other divided parts 141. The XY plane is perpendicular to the rotation axis (rotation center C1). Furthermore, the size of each divided part 141 may be changed.

[0057] In the superimposition process 74, a superimposed image P4 is generated by superimposing the converted image P3 on the second image P2. That is, after the converted image P3 as shown in Fig. 4C is generated by the image conversion process, the converted image P3 is superimposed on the second image P2 (see Fig. 3) to generate the superimposed image P4 as shown in Fig. 5.

[0058] Here, the converted image P3 is an image obtained by processing the defective portion 14 of the first object 1. Therefore, the object 4 shown in the superimposed image P4 includes a defective portion 44 that was created by processing the defective portion 14. In other words, the superimposed image P4 is an image (defective product image) in which at least the defect occurrence location E4 of the object 4 is displayed.

[0059] The superimposed image P4 is a three-dimensional image that displays depth as gradation, i.e., it contains information on the coordinates in the depth direction of each of the converted image P3 and the second image P2 as gradation.

[0060] The superimposition process 74 preferably includes an interpolation process that interpolates the boundary between the converted image P3 and the second image P2 so as to smoothly connect the converted image P3 and the second image P2. This allows a more natural superimposed image P4 to be generated. The interpolation process is realized, for example, by linear interpolation.

[0061] (6) Setting information Next, we will explain the setting information 81 that defines the processing performed by the image processing unit 53. The setting information 81 is information related to the processing of generating the superimposed image P4 from the first image P1 and the second image P2. More specifically, the setting information 81 includes information related to at least one of the extraction processing 71, the image conversion processing (processes 72 and 73), and the superimposition processing 74.

[0062] The setting information input unit 54 acquires the setting information 81. The setting information input unit 54 acquires the setting information 81 from a user interface 55 that accepts a user's input operation of the setting information 81. The user interface 55 includes, for example, at least one of a mouse, a keyboard, a touchpad, and the like.

[0063] The setting information 81 includes one or more of the following information: The first type of information that the setting information 81 may include is information for determining the range of the extracted portion 140 in the first image P1. The second type of information that the setting information 81 may include is information for determining the number of divisions of the extracted portion 140 in the image conversion process. The third type of information that the setting information 81 may include is information for determining changes in at least one of the gradation, position, size, and orientation of the extracted portion 140 in the image conversion process. The fourth type of information that the setting information 81 may include is information for determining the mixing ratio of the converted image P3 and the second image P2 in the superimposition process 74. The fifth type of information that the setting information 81 may include is information regarding at least one of the gradation, position, size, and orientation (rotation angle) of each of the multiple divided parts 141. The sixth type of information that the setting information 81 may include is information regarding the expansion / contraction rate used in the process of expanding or compressing the extracted portion 140 in a predetermined direction.

[0064] The image processing unit 53 can, for example, define the range selected by the user using the user interface 55 as the range of the extracted portion 140. The image processing unit 53 can define the number of divisions input by the user using the user interface 55 as the number of divisions of the extracted portion 140 and generate the same number of divided parts 141. The image processing unit 53 can also change the gradation, position, size, and orientation of each divided part 141 or the entire extracted portion 140 according to information input by the user using the user interface 55. The image processing unit 53 can also superimpose the converted image P3 and the second image P2 using a blending ratio input by the user using the user interface 55. The blending ratio is, for example, the α value of each of the converted image P3 and the second image P2. The image processing unit 53 superimposes the converted image P3 on the second image P2 using α blending.

[0065] The setting information generation unit 56 generates the setting information 81. The setting information generation unit 56 determines the range of the extracted portion 140 in the first image P1, for example, using a predetermined trained model for detecting the defective portion 14 (extracted portion 140) from an image. Furthermore, the setting information generation unit 56 determines the number of divisions of the extracted portion 140, for example, depending on the size of the extracted portion 140.

[0066] Note that randomness may be imparted to the setting information 81 generated by the setting information generating unit 56. For example, the setting information generating unit 56 may randomly determine at least one of the position, size, and orientation of the extracted portion 140 (converted image P3) relative to the second object 2.

[0067] The term "random" does not necessarily mean that all events occur with equal probability. For example, the setting information generator 56 may refer to information on the probability of defect occurrence in each of a plurality of regions on the second object 2, and set parameters for randomly determining the position of the extraction portion 140 so that a region with a higher defect occurrence probability is more likely to be selected as the position of the extraction portion 140.

[0068] Furthermore, the setting information generating unit 56 may assign a label to the superimposed image P4. For example, the setting information generating unit 56 may determine the label of the superimposed image P4 according to the label assigned to the first image P1. Specifically, if the label assigned to the first image P1 is "defective product" and the extracted portion 140 in the first image P1 matches the location E1 where the defect has occurred, the setting information generating unit 56 may set the label of the superimposed image P4 to "defective product." In this case, the label may include the type of defect, and the type of defect may be the same as the type of defect in the first image P1.

[0069] (7) Display output unit and display device The display output unit 57 outputs the information to the display device 58. The display device 58 displays an image according to the information received from the display output unit 57.

[0070] The display device 58 includes a display. The display is, for example, a liquid crystal display or an organic electroluminescence (EL) display. The display device 58 displays, for example, a first image P1, a second image P2, a converted image P3, and a superimposed image P4. The display device 58 is used as an output interface that displays setting information 81 when a user inputs setting information 81 to the user interface 55. For example, when the first image P1 is displayed on the display device 58, the user can determine the range of the extracted portion 140 by operating the user interface 55 to move a cursor displayed on the display device 58. Furthermore, the user can input the gradation and orientation of the extracted portion 140 or the α values ​​of the converted image P3 and the second image P2, for example, by aligning the cursor with a slider displayed on the display device 58 and dragging it.

[0071] Furthermore, the display output unit 57 outputs information relating to the plurality of divided parts 141 to the display device 58. The display device 58 displays the plurality of divided parts 141 based on the information relating to the plurality of divided parts 141 acquired from the display output unit 57. This allows the user to select a desired divided part 141 and input setting information 81 for the selected divided part 141.

[0072] The display device 58 also displays a determination result 83 output from a determination unit 61, which will be described later.

[0073] (8) Types of defects In this embodiment, the first object 1 is an article formed by welding two or more base materials (here, a first metal plate 11 and a second metal plate 12). In other words, the first object 1 is a welded product.

[0074] Types of defects that occur in weldments include, for example, pits, spatters, protrusions, holes, and undercuts. The extracted portion 140 includes at least a portion of a defect occurrence location E1 related to at least one of the pits, spatters, protrusions, holes, and undercuts that occur in the weldment. In this embodiment, the extracted portion 140 includes the entire defect occurrence location E1, and the defect is an undercut. The gradation of the extracted portion 140 corresponds to the depth of the undercut.

[0075] A pit is a depression that occurs in the bead 13. A spatter is a spherical, conical, or other protrusion that occurs in the bead 13. A protrusion is a cylindrical protrusion that occurs in the bead 13. A hole is a portion of the bead 13 that has melted and is missing. An undercut is a depression that occurs around the bead 13.

[0076] The extracted portion 140 is not limited to the defective portion 14, and may be a portion where no defect occurs. Alternatively, the extracted portion 140 may include the defective portion 14 and a portion in the vicinity of the defective portion 14. Furthermore, the extracted portion 140 may include a plurality of defective portions 14.

[0077] In particular, when the defect occurring in the first object 1 is a defect occurring in a small area (such as a pit, spatter, or protrusion), it is preferable that the extracted portion 140 includes not only one defective portion 14, but also a portion adjacent to the defective portion 14 or another defective portion 14. This ensures the length of the extracted portion 140, and makes the process of bending the extracted portion 140 and the process of dividing the extracted portion 140 more effective.

[0078] (9) Machine learning and pass / fail judgment The image output unit 59 outputs the superimposed image P4 generated by the image processing unit 53 to the learning unit 60. The learning unit 60 performs machine learning using the superimposed image P4 as learning data (learning data set). In this way, the learning unit 60 generates a trained model 82. As described above, the learning unit 60 may use not only the superimposed image P4 but also the first image P1 and the second image P2 as learning data.

[0079] The training dataset is generated by assigning labels to multiple image data indicating "good" or "defective," and in the case of defective products, the type and location of the defect. The labeling work is performed, for example, by a user via a user interface 55 or by the setting information generating unit 56. The learning unit 60 generates a trained model 82 by performing machine learning on the state of the object (good state, defective state, type of defect, location of defect, etc.) using the training dataset.

[0080] The learning unit 60 may perform re-learning using a learning dataset including newly acquired learning data, thereby improving the performance of the trained model 82. For example, if a new type of defect is found in an object, the learning unit 60 may be made to perform re-learning on the new defect.

[0081] In a factory production line, for example, an imaging device captures an image of an object and generates an inspection image P5. More specifically, the imaging device captures an image of an object on which a bead has been formed after the actual welding process, and generates the inspection image P5. The inspection image acquisition unit 62 acquires the inspection image P5 from the imaging device. The determination unit 61 uses the trained model 82 generated by the learning unit 60 to perform a pass / fail determination on the inspection image P5 (object) acquired by the inspection image acquisition unit 62. If the object is defective, the determination unit 61 determines the type of defect and the location of the defect. The determination unit 61 outputs a determination result 83. The determination result 83 is output to, for example, the display device 58, which displays the determination result 83. A user can check the determination result 83 via the display device 58. The production equipment may be controlled so that an object determined to be "defective" by the determination unit 61 is discarded before being transported to the next process. Furthermore, the determination result 83 is output to, for example, a data server and stored in the data server.

[0082] As described above, the image generation system 5 includes an inspection image acquisition unit 62 that acquires an inspection image P5, and a determination unit 61 that determines whether the inspection image P5 is good or bad using a trained model 82. The trained model 82 is generated based on the superimposed image P4 generated by the image processing unit 53.

[0083] (10) Operation flow Next, the flow of processing by which the image generation system 5 generates the superimposed image P4 will be described with reference to Fig. 6. Note that the flow shown in Fig. 6 is merely an example, and the order of processing may be changed as appropriate, and processing may be added or omitted as appropriate.

[0084] The setting information 81 acquired in steps ST2 to ST5 and ST7 may be input by the user via the user interface 55, or may be generated by the setting information generation unit 56. Also, a portion of the setting information 81 may be input by the user via the user interface 55, and another portion of the setting information 81 may be generated by the setting information generation unit 56.

[0085] First, the first image acquisition unit 51 acquires the first image P1 (step ST1). Next, the image processing unit 53 acquires setting information 81 related to the extraction settings and performs the extraction process 71 based on the extraction settings (step ST2). As a result, an extraction image is generated (see FIG. 4A). The setting information 81 related to the extraction settings is information for determining the range of the extraction region 140 in the first image P1.

[0086] Next, image processing unit 53 acquires setting information 81 related to the transformation settings and performs transformation processing based on the transformation settings (step ST3). As a result, a transformed image is generated (see FIG. 4B). Setting information 81 related to the transformation settings is information referenced for transforming extracted portion 140 so as to bend it, and includes, for example, information indicating the bending location and bending angle.

[0087] Next, the image processing unit 53 acquires the setting information 81 regarding the division setting and performs the division process based on the division setting (step ST4). The setting information 81 regarding the division setting includes, for example, information for determining the number of divisions of the extracted part 140.

[0088] Next, image processing unit 53 acquires setting information 81 related to the conversion settings and performs conversion processing based on the conversion settings (step ST5). Setting information 81 related to the conversion settings is information for determining a change in at least one of the gradation of extracted portion 140, the position of extracted portion 140, the size of extracted portion 140, and the orientation of extracted portion 140. The conversion processing is processing that changes at least one of the gradation of extracted portion 140, the position of extracted portion 140, the size of extracted portion 140, and the orientation of extracted portion 140. Furthermore, the conversion processing can change the gradation, position, size, and orientation (rotation angle) for each divided part 141.

[0089] Next, the second image acquisition unit 52 acquires the second image P2 (step ST6). The image processing unit 53 also acquires setting information 81 related to the superimposition setting and performs the superimposition process 74 based on the superimposition setting (step ST7). The setting information 81 related to the superimposition setting includes, for example, information for determining the mixture ratio of the converted image P3 and the second image P2 in the superimposition process 74.

[0090] Thereafter, the image processing unit 53 outputs the superimposed image P4 generated by the superimposition process 74 (step ST8).

[0091] The image processing unit 53 determines whether or not a prescribed number (or more than the prescribed number) of superimposed images P4 have been output (step ST9). If the prescribed number of superimposed images P4 have been output, the image processing unit 53 ends the process of generating the superimposed images P4. On the other hand, if the prescribed number of superimposed images P4 have not been output, the process returns to step ST1. Steps ST1 to ST8 are repeated to generate the prescribed number of superimposed images P4.

[0092] The minimum required number of first images P1 is 1, and the minimum required number of second images P2 is 1. At least one of the first image P1 and the second image P2 may be changed when generating one superimposed image P4 and when generating another superimposed image P4.

[0093] Furthermore, a certain superimposed image P4 generated by the image processing unit 53 may be used as the first image P1 or the second image P2 when another superimposed image P4 is generated.

[0094] (11) Advantages The image generation system 5 of this embodiment can generate a superimposed image P4 suitable for machine learning. In particular, because the extracted portion 140 is deformed to conform to the shape of the second object 2 in the second image P2, the superimposed image P4 can be made more natural than when an image of the extracted portion 140 is superimposed on the second image P2 without such deformation. In other words, defects that actually occur in the second object 2 can be reproduced in the superimposed image P4. Therefore, the trained model 82 generated using the superimposed image P4 can be a model that can accurately determine the presence or absence of defects and the type of defect.

[0095] Furthermore, the image of the extracted portion 140 is processed for each divided part 141. This makes it easier to determine the parameters (setting information 81) that determine the processing for processing the image of the extracted portion 140, compared to when the extracted portion 140 is not divided, and makes it easier to generate the superimposed image P4 desired by the user, etc. Therefore, it is possible to improve the accuracy of machine learning using the superimposed image P4.

[0096] (Variation 1) Modification 1 will be described below with reference to Figures 7A to 7E. The configuration of image generation system 5 is common to the above-described embodiment and Modification 1, so the same components as those in the embodiment are denoted by the same reference numerals and description thereof will be omitted.

[0097] As in the embodiment, the setting information 81 related to the process of generating the superimposed image P4 from the first image P1 and the second image P2 may be input by the user via the user interface 55, or may be generated by the setting information generation unit 56. Alternatively, a portion of the setting information 81 may be input by the user via the user interface 55, and another portion of the setting information 81 may be generated by the setting information generation unit 56.

[0098] In this variant example 1, the image processing unit 53 performs a division process to divide the extracted portion 140 into a plurality of divided parts 141, and then performs a process to deform the extracted portion 140 to conform to the shape of the second object 2 in the second image P2.

[0099] First, as in the embodiment, the image processing unit 53 performs processing to extract the extracted portion 140 from the first object 1 shown in the first image P1 (see FIG. 2). As a result, an extracted image in which the extracted portion 140 has been extracted is generated, as shown in FIG. 7A. Furthermore, the image processing unit 53 may also perform processing to invert (mirror-invert) the extracted portion 140 as appropriate.

[0100] Next, image processing unit 53 performs an expansion / contraction process to expand or contract extracted portion 140 in a predetermined direction. As a result, a converted image P3 as shown in Fig. 7B is generated. The expansion / contraction process may be omitted as appropriate.

[0101] Next, the image processing unit 53 performs a division process, thereby generating a converted image P3 as shown in Fig. 7C.

[0102] Furthermore, the image processing unit 53 processes the image of the extracted portion 140 for each of the divided parts 141. The image processing unit 53 performs at least a process of deforming the extracted portion 140 so that it conforms to the shape of the second object 2 in the second image P2 (hereinafter referred to as a bending process).

[0103] The rotation center C1 of each of the multiple divided parts 141 is located adjacent to other divided parts 141 (see FIG. 7E). In the bending process, at least some of the divided parts 141 are rotated around their respective rotation centers C1, thereby deforming the extracted portion 140. The bending process generates a transformed image P3 as shown in FIG. 7D.

[0104] The bending process may include a process of correcting the converted image P3 after rotating at least some of the divided parts 141. For example, the converted image P3 may be corrected so that the multiple divided parts 141 are smoothly joined together.

[0105] Furthermore, the image processing unit 53 performs a conversion process (process 73) on at least some of the divided parts 141. The conversion process is a process that changes at least one of the gradation of the extracted portion 140, the position of the extracted portion 140, the size of the extracted portion 140, and the orientation of the extracted portion 140. Here, the conversion process can change the gradation, position, size, and orientation for each divided part 141.

[0106] After the conversion process, the image processing unit 53 superimposes the converted image P3 on the second image P2 to generate a superimposed image P4 as shown in FIG.

[0107] As described above, in this modification, the bending process is achieved by dividing the extracted region 140 into a plurality of divided parts 141 and rotating at least some of the divided parts 141. Therefore, even when the user determines and inputs the setting information 81 related to the bending process, the amount of information required to determine the setting information 81 can be reduced, and the setting information 81 can be easily determined.

[0108] (Other Modifications of the Embodiments) Other variations of the embodiment are listed below. The following variations may be implemented in appropriate combination. The following variations may also be implemented in appropriate combination with the above-described variation 1.

[0109] The division process of dividing the extracted portion 140 into a plurality of divided parts 141 is not essential. If the division process is not performed, for example, information such as the gradation of each coordinate of the extracted portion 140 may be changed for each pixel, or may be changed uniformly across the entire extracted portion 140.

[0110] The trained model 82 is not limited to a model used for welding visual inspection, but may be a model used for various types of inspection, etc. Furthermore, the trained model 82 is not limited to a model used for inspection, but may be a model used for various types of image recognition.

[0111] The entity that executes the extraction process 71 is not limited to the image processing unit 53, but may be a configuration external to the image generation system 5.

[0112] At least one of the user interface 55, the display device 58, the learning unit 60, and the determining unit 61 may be configured external to the image generating system 5.

[0113] The display device 58 may be a mobile terminal such as a smartphone or a tablet terminal.

[0114] The images handled by the image generation system 5 (first image P1, second image P2, converted image P3, superimposed image P4, and inspection image P5, etc.) are not limited to three-dimensional images, but may be two-dimensional images or images with four or more dimensions.

[0115] The superimposed image P4 generated from the first image P1 and the second image P2 is not limited to a defective product image, but may also be a non-defective product image.

[0116] The first object 1 in the first image P1 may be an article of the same shape as the second object 2 in the second image P2, or may be an article of a different shape.

[0117] The first image P1 may be the same image as the second image P2.

[0118] The first image P1 and the second image P2 may be images of a defective product or images of a non-defective product, or one of the first image P1 and the second image P2 may be an image of a non-defective product and the other an image of a defective product.

[0119] The first image P1 may be an image of a part of the first object 1. The second image P2 may be an image of a part of the second object 2.

[0120] The image processing unit 53 may extract a plurality of extracted portions 140. In this case, a plurality of converted images P3 are generated.

[0121] The image processing unit 53 may generate a plurality of converted images P3 from one extracted portion 140.

[0122] In the superimposition process 74, the image processing unit 53 may superimpose a plurality of converted images P3 onto the second image P2.

[0123] In the converted image P3, it is not essential that the multiple divided parts 141 are connected to each other, and some divided parts 141 may be provided apart from other divided parts 141.

[0124] The rotation center C1 of the divided part 141 may be located at a position away from the divided part 141.

[0125] The first image P1, the second image P2, the converted image P3, the superimposed image P4, and the inspection image P5 may be luminance image data that represents the luminance of an object using gradation.

[0126] In the embodiment, the "tone" refers to the depth of a single color (for example, black). Alternatively, the "tone" may refer to the depth of each of multiple colors (for example, the three RGB colors).

[0127] The range in the second image P2 in which the converted image P3 is to be placed may be set by the user operating the user interface 55. For example, the range in the second image P2 in which the converted image P3 is to be placed may be selectable from among the first metal plate 21, the second metal plate 22, and the bead 23. When the first metal plate 21 is selected as the range in the second image P2 in which the converted image P3 is to be placed, the image processing unit 53 may place the converted image P3 at a randomly determined position on the first metal plate 21, for example.

[0128] The image generation system 5 or the image generation method according to the present disclosure includes a computer system. The computer system is primarily composed of a processor and memory as hardware. At least a portion of the functions of the image generation system 5 or the image generation method according to the present disclosure are realized by the processor executing a program stored in the memory of the computer system. The program may be pre-stored in the memory of the computer system, provided via a telecommunications line, or provided on a non-transitory recording medium readable by the computer system, such as a memory card, optical disk, or hard disk drive. The processor of the computer system is composed of one or more electronic circuits, including a semiconductor integrated circuit (IC) or a large-scale integrated circuit (LSI). The integrated circuits, such as ICs and LSIs, are referred to by different names depending on the degree of integration, and include integrated circuits called system LSIs, very large-scale integrations (VLSIs), or ultra-large-scale integrations (ULSIs). Furthermore, field-programmable gate arrays (FPGAs), which are programmable after the LSI is manufactured, or logic devices capable of reconfiguring the connections within the LSI or the circuit partitions within the LSI, can also be used as processors. The electronic circuits may be integrated into one chip or distributed across multiple chips. The chips may be integrated into one device or distributed across multiple devices. The computer system referred to here includes a microcontroller having one or more processors and one or more memories. Therefore, the microcontroller is also composed of one or more electronic circuits including a semiconductor integrated circuit or a large-scale integrated circuit.

[0129] Furthermore, it is not essential for the image generation system 5 that multiple functions are integrated into one device, and the components of the image generation system 5 may be distributed across multiple devices. Furthermore, at least some of the functions of the image generation system 5, for example, at least some of the functions of at least one of the image processing unit 53, the learning unit 60, and the determination unit 61, may be realized by a server or the cloud (cloud computing), etc. Conversely, multiple functions of the image generation system 5 may be integrated into one device.

[0130] (summary) The above-described embodiments and the like disclose the following aspects.

[0131] The image generation system (5) according to the first aspect includes a first image acquisition unit (51), a second image acquisition unit (52), and an image processing unit (53). The first image acquisition unit (51) acquires a first image (P1) obtained by capturing an image of a first object (1). The second image acquisition unit (52) acquires a second image (P2) obtained by capturing an image of a second object (2). The image processing unit (53) performs image conversion processing and superimposition processing (74). The image conversion processing is processing for generating a converted image (P3) that is an image obtained by processing a predetermined extracted portion (140) of the first object (1) based on the first image (P1). The superimposition processing (74) is processing for generating a superimposed image (P4) by superimposing the converted image (P3) on the second image (P2). The image transformation process includes a process of transforming the extracted portion (140) to conform to the shape of the second object (2) in the second image (P2).

[0132] According to the above configuration, it is possible to generate an image (superimposed image (P4)) suitable for machine learning. In particular, before generating the superimposed image (P4) by superimposing the transformed image (P3) on the second image (P2), a process is performed to deform the extracted portion (140) according to the shape of the second object (2) in the second image (P2), and the transformed image (P3) is generated. As a result, compared to when such a process is not performed, the superimposed image (P4) is more likely to resemble an actual image, not an image generated by superimposition. Therefore, it is possible to improve the accuracy of machine learning using the superimposed image (P4).

[0133] In the image generation system (5) according to the second aspect, in the first aspect, the image conversion process further includes a process of dividing the extracted portion (140) into a plurality of divided parts (141).

[0134] According to the above configuration, by dividing the extraction part (140), it becomes easier to set up the extraction part (140).

[0135] In addition, in the image generation system (5) according to the third aspect, in the first or second aspect, the image conversion process includes a process of changing at least one of the gradation of the extracted portion (140), the position of the extracted portion (140), the size of the extracted portion (140), and the orientation of the extracted portion (140).

[0136] According to the above configuration, a variety of superimposed images (P4) can be generated.

[0137] Furthermore, the image generation system (5) according to a fourth aspect is any one of the first to third aspects, and further includes a setting information input unit (54). The setting information input unit (54) acquires setting information (81) related to a process of generating a superimposed image (P4) from a first image (P1) and a second image (P2). The setting information input unit (54) acquires the setting information (81) from a user interface (55) that accepts a user's input operation of the setting information (81).

[0138] According to the above configuration, the user can specify the setting information (81).

[0139] In addition, the image generation system (5) according to a fifth aspect is any one of the first to fourth aspects, and further includes a setting information generation unit (56). The setting information generation unit (56) generates setting information (81) related to a process of generating a superimposed image (P4) from the first image (P1) and the second image (P2).

[0140] According to the above configuration, the setting information (81) is generated even if the user does not specify the setting information (81).

[0141] In the image generation system (5) according to a sixth aspect, in the fourth or fifth aspect, the setting information (81) includes one or more of the following first to fourth pieces of information. The first piece of information that the setting information (81) may include is information for determining the range of the extracted portion (140) in the first image (P1). The second piece of information that the setting information (81) may include is information for determining the number of divisions of the extracted portion (140) in the image conversion process. The third piece of information that the setting information (81) may include is information for determining at least one change in the gradation of the extracted portion (140), the position of the extracted portion (140), the size of the extracted portion (140), and the orientation of the extracted portion (140) in the image conversion process. The fourth piece of information that the setting information (81) may include is information for determining the mixing ratio of the converted image (P3) and the second image (P2) in the superimposition process (74).

[0142] According to the above configuration, the converted image (P3) can be generated in accordance with the contents of the setting information (81).

[0143] In addition, in an image generation system (5) according to a seventh aspect, in any one of the first to sixth aspects, the first image (P1) is an image of a defect occurrence location (E1) of a first object (1). The second image (P2) is an image of a non-defective second object (2). The extracted portion (140) includes at least a part of the defect occurrence location (E1).

[0144] According to the above configuration, a defective product image can be generated as the superimposed image (P4).

[0145] In an image generation system (5) according to an eighth aspect, in the seventh aspect, the first object (1) is a weldment, and the extracted portion (140) includes at least a part of a defect occurrence location (E1) related to at least one of pits, spatters, protrusions, holes, and undercuts that occur in the weldment.

[0146] According to the above configuration, an image of a defective welded product can be generated as the superimposed image (P4).

[0147] In addition, the image generation system (5) according to a ninth aspect is any one of the first to eighth aspects, and further includes an image output unit (59). The image output unit (59) outputs the superimposed image (P4) generated by the image processing unit (53) to a learning unit (60). The learning unit (60) performs machine learning using the superimposed image (P4) as learning data.

[0148] According to the above configuration, a highly accurate trained model (82) can be generated by machine learning based on the superimposed image (P4).

[0149] In addition, the image generation system (5) according to a tenth aspect is any one of the first to ninth aspects, and further includes an inspection image acquisition unit (62) and a determination unit (61). The inspection image acquisition unit (62) acquires an inspection image (P5). The determination unit (61) determines whether the inspection image (P5) is good or bad using a trained model (82). The trained model (82) is generated based on a superimposed image (P4) generated by an image processing unit (53).

[0150] According to the above configuration, by using the trained model (82) generated based on the superimposed image (P4), it is possible to perform a highly accurate pass / fail judgment.

[0151] In addition, in the image generation system (5) according to the eleventh aspect, in any one of the first to tenth aspects, in the image conversion process, deforming the extracted portion (140) to conform to the shape of the second object (2) in the second image (P2) means deforming the extracted portion (140) to conform to the contour or edge of at least a part of the second object (2) in the second image (P2).

[0152] With the above arrangement, the extracted region (140) can be modified using known techniques for extracting contours or edges.

[0153] In addition, in the image generation system (5) according to the twelfth aspect, in any one of the first to eleventh aspects, the superimposed image (P4) is a three-dimensional image in which depth is displayed as gradation.

[0154] According to the above configuration, machine learning can be performed taking depth information into consideration.

[0155] The configurations other than the first aspect are not essential for the image generation system (5) and can be omitted as appropriate.

[0156] Furthermore, an image generating method according to a thirteenth aspect includes a first image acquisition process, a second image acquisition process, an image conversion process, and a superimposition process (74). The first image acquisition process acquires a first image (P1) obtained by capturing a first object (1). The second image acquisition process acquires a second image (P2) obtained by capturing a second object (2). The image conversion process is a process of generating a converted image (P3) that is an image obtained by processing a predetermined extracted portion (140) of the first object (1) based on the first image (P1). The superimposition process (74) is a process of generating a superimposed image (P4) by superimposing the converted image (P3) on the second image (P2). The image conversion process includes a process of deforming the extracted portion (140) to conform to the shape of the second object (2) in the second image (P2).

[0157] According to the above configuration, it is possible to generate an image (superimposed image (P4)) suitable for machine learning.

[0158] In addition, the image generation method according to the 14th aspect further includes, in the 13th aspect, a setting information generation process for generating setting information (81) related to a process for generating a superimposed image (P4) from a first image (P1) and a second image (P2).

[0159] According to the above configuration, the setting information (81) is generated even if the user does not specify the setting information (81).

[0160] The configurations other than those of the thirteenth aspect are not essential for the image generating method and can be omitted as appropriate.

[0161] A program according to a fifteenth aspect is a program for causing one or more processors of a computer system to execute the image generating method according to the thirteenth or fourteenth aspect.

[0162] According to the above configuration, it is possible to generate an image (superimposed image (P4)) suitable for machine learning.

[0163] Not limited to the above aspects, various configurations (including modified examples) of the image generation system (5) according to the embodiment can be embodied as an image generation method, a (computer) program, or a non-transitory recording medium on which a program is recorded. [Explanation of symbols]

[0164] 1. First Object 2 Second Object 5. Image Generation System 51 First image acquisition unit 52 Second image acquisition unit 53 Image processing section 54 Setting information input section 55 User Interface 56 Configuration information generation unit 59 Image output unit 60 Learning Department 61 Judgment section 62 Inspection image acquisition unit 74 Superimposition Processing 81 Setting information 82 trained models 140 Extraction site 141 Separate Parts E1 Defect location P1 First image P2 Second image P3 converted image P4 Superimposed image P5 Inspection image

Claims

1. a first image acquisition unit that acquires a first image obtained by capturing an image of a first object; a second image acquisition unit that acquires a second image obtained by capturing an image of a second object; an image processing unit that performs an image conversion process to generate a converted image that is an image obtained by processing a predetermined extracted portion of the first object based on the first image, and a superimposition process to generate a superimposed image by superimposing the converted image on the second image, the image transformation process includes a process of transforming the extracted portion to conform to a shape of the second object in the second image, and a process of dividing the extracted portion into a plurality of divided parts. Image generation system.

2. A first image acquisition unit that acquires a first image obtained by capturing an image of a first object; a second image acquisition unit that acquires a second image obtained by capturing an image of a second object; an image processing unit that performs an image conversion process to generate a converted image that is an image obtained by processing a predetermined extracted portion of the first object based on the first image, and a superimposition process to generate a superimposed image by superimposing the converted image on the second image, the image transformation process includes a process of transforming the extracted portion to conform to a shape of the second object in the second image; In the image conversion process, deforming the extracted portion to conform to the shape of the second object in the second image means deforming the extracted portion to conform to a contour or an edge of at least a part of the second object in the second image. Image generation system.

3. the image conversion process includes a process of changing at least one of a gradation of the extracted portion, a position of the extracted portion, a size of the extracted portion, and an orientation of the extracted portion; 3. The image generation system according to claim 1 or 2.

4. a setting information input unit that acquires setting information related to a process of generating the superimposed image from the first image and the second image; the setting information input unit acquires the setting information from a user interface that accepts an input operation of the setting information by a user; 3. The image generation system according to claim 1 or 2.

5. further comprising a setting information generating unit that generates setting information related to a process of generating the superimposed image from the first image and the second image; 3. The image generation system according to claim 1 or 2.

6. The setting information is information for determining the range of the extracted portion in the first image; information for determining the number of divisions of the extracted portion in the image conversion processing; information for determining a change in at least one of the gradation of the extracted portion, the position of the extracted portion, the size of the extracted portion, and the orientation of the extracted portion in the image conversion process; and and information for determining a mixing ratio of the converted image and the second image in the superimposition process. The image generation system of claim 4 .

7. the first image is an image of a location where a defect has occurred in the first object, the second image is an image obtained by capturing the second object as a non-defective product, the extracted portion includes at least a part of the location where the defect occurs; 3. The image generation system according to claim 1 or 2.

8. the first object is a weldment; The extracted portion includes at least a portion of a defect occurring in the weldment related to at least one of pits, spatters, protrusions, holes, and undercuts. The image generation system of claim 7.

9. an image output unit that outputs the superimposed image generated by the image processing unit to a learning unit that performs machine learning using the superimposed image as learning data; 3. The image generation system according to claim 1 or 2.

10. an inspection image acquisition unit that acquires an inspection image; and a determination unit that determines whether the inspection image is good or bad using a trained model generated based on the superimposed image generated by the image processing unit.

3. The image generation system according to claim 1 or 2.

11. In the image conversion process, deforming the extracted portion to conform to the shape of the second object in the second image means deforming the extracted portion to conform to a contour or an edge of at least a part of the second object in the second image. The image generation system of claim 1 .

12. The superimposed image is a three-dimensional image in which depth is displayed as gradation.

3. The image generation system according to claim 1 or 2.

13. a first image acquisition process for acquiring a first image obtained by capturing an image of a first object; a second image acquisition process for acquiring a second image obtained by capturing an image of a second object; an image conversion process for generating a converted image, which is an image obtained by processing a predetermined extracted portion of the first object, based on the first image; a superimposition process for generating a superimposed image by superimposing the converted image on the second image, the image transformation process includes a process of transforming the extracted portion to conform to a shape of the second object in the second image, and a process of dividing the extracted portion into a plurality of divided parts. Image generation method.

14. A first image acquisition process for acquiring a first image capturing a first object; a second image acquisition process for acquiring a second image obtained by capturing an image of a second object; an image conversion process for generating a converted image, which is an image obtained by processing a predetermined extracted portion of the first object, based on the first image; a superimposition process for generating a superimposed image by superimposing the converted image on the second image, the image transformation process includes a process of transforming the extracted portion to conform to a shape of the second object in the second image; In the image conversion process, deforming the extracted portion to conform to the shape of the second object in the second image means deforming the extracted portion to conform to a contour or an edge of at least a part of the second object in the second image. Image generation method.

15. The method further includes a setting information generation process for generating setting information related to a process for generating the superimposed image from the first image and the second image.

15. The image generating method according to claim 13 or 14.

16. For causing one or more processors of a computer system to execute the image generation method according to claim 13 or 14, program.

Images