Training data acquisition apparatus and training data acquisition method

Abstract

The present invention causes a processor 22 to execute at least one of: a step for generating, on the basis of first and second 3D data for synthesis representing a point cloud including a weld defect recessed in the Z-axis direction, synthesized 3D data representing a 3D image in which a bottom of the weld defect is formed at a certain depth in a region on an XY coordinate plane where the bottom of the weld defect is located in the first 3D data for synthesis and in a region on an XY coordinate plane where the bottom of the weld defect is located in the second 3D data for synthesis; and a step for generating, on the basis of first and second 3D data for synthesis representing a point cloud including a weld defect protruding in the Z-axis direction, synthesized 3D data representing a 3D image in which a top of the weld defect is formed at a certain protrusion height in a region on an XY coordinate plane where the top of the weld defect is located in the first 3D data for synthesis and in a region on an XY coordinate plane where the top of the weld defect is located in the second 3D data for synthesis.

Description

Learning data acquisition device and learning data acquisition method

[0001] The present disclosure relates to a learning data acquisition device and a learning data acquisition method for acquiring image data used in machine learning.

[0002] The training data acquisition device disclosed in Patent Document 1 acquires training data used in machine learning for a machine learning system that automatically generates output images from input images. The training data acquisition device acquires, as training data, pseudo sample images used as input images for the machine learning system during machine learning and pseudo-labeled images used as output images for the machine learning system during machine learning. The training data acquisition device first generates a large number of pseudo-labeled images by inputting random values ​​into an image generation engine that has performed machine learning on multiple original-labeled images. The training data acquisition device then generates the pseudo sample images based on these pseudo-labeled images in accordance with the conversion characteristics from the original-labeled images to the original sample images. This enables a large amount of training data to be acquired in a short period of time.

[0003] JP 2019-46269 A

[0004] It is conceivable to use machine learning to create a model that outputs welding defect information, including the location of welding defects, based on 3D data representing 3D images. Creating such a model requires training data consisting of multiple data sets: 3D data representing images of the area around a welding mark captured by a 3D sensor, and welding defect information to be output based on the 3D data. One conceivable method for creating the 3D data constituting the training data is to acquire 3D images of welding marks that do not include welding defects and 3D images of welding defects by measurement with a 3D sensor, process them, and combine them. However, this method requires the preparation of a large number of 3D images of welding defects, which is time-consuming and requires a long work time.

[0005] Furthermore, even if a 3D image of a welding defect is generated using an image generation engine such as that of Patent Document 1, the above-mentioned problem cannot be solved because a 3D image of the welding defect is required for machine learning in the image generation engine.

[0006] The present disclosure has been made in consideration of these points, and its purpose is to reduce the effort and work time required to acquire learning data.

[0007] In order to achieve the above object, the present disclosure provides a learning data acquisition device for acquiring learning data to be used in machine learning for creating a model, the model outputting welding defect information including the position of the welding defect based on 3D data representing a 3D image, the learning data acquisition device including: a first synthesis step of generating synthesized 3D data representing a 3D image in which the bottom of the welding defect is formed at a certain depth in an area on an XY coordinate plane where the bottom of the welding defect is located in the first synthesis 3D data and in an area on an XY coordinate plane where the bottom of the welding defect is located in the second synthesis 3D data, based on first and second synthesis 3D data representing a point cloud including the welding defect recessed in the Z-axis direction in a predetermined XYZ orthogonal coordinate system and having common image sizes in the X-axis direction and the Y-axis direction; and a second synthesis step of generating, based on first and second synthesis 3D data that represent a point cloud including a welding defect protruding toward a target surface, 3D data representing a 3D image in which the top of the welding defect is formed at a certain protruding height in an area on an XY coordinate plane where the top of the welding defect is located in the first synthesis 3D data and in an area on an XY coordinate plane where the top of the welding defect is located in the second synthesis 3D data, the area being based on first and second synthesis 3D data that represent a point cloud including a welding defect protruding toward a target surface, the image sizes in the X-axis direction and the Y-axis direction being common to each other; and a learning data creation step of creating a plurality of learning 3D data by combining a plurality of patterns of 3D images of the welding defects represented by the plurality of defect 3D data including final 3D data based on the synthesis 3D data.

[0008] This allows the number of 3D images of welding defects used to create 3D learning data to be increased by acquiring final 3D data based on at least two sets of 3D data to be combined, without increasing the number of measurements by the 3D sensor, thereby reducing the effort and time required to acquire learning data.

[0009] Furthermore, when the learning data acquisition process executes the first synthesis step, a bottom of a certain depth is formed in the synthesized 3D data in the region on the XY coordinate plane where the bottom of the weld defect is located in the first synthesis 3D data and in the region on the XY coordinate plane where the bottom of the weld defect is located in the second synthesis 3D data. That is, a step is not formed on the bottom surface of the weld defect shown in the synthesized 3D data due to the difference in depth of the bottom of the weld defect shown in the first and second synthesis 3D data. Therefore, it is possible to prevent the prediction accuracy of the model from deteriorating due to the step being formed on the bottom surface of the weld defect shown in the synthesized 3D data.

[0010] Furthermore, when the learning data acquisition process executes the second synthesis step, in the synthesized 3D data, a peak of a certain protruding height is formed in the region on the XY coordinate plane where the peak of the weld defect is located in the first synthesis 3D data and in the region on the XY coordinate plane where the peak of the weld defect is located in the second synthesis 3D data. That is, a step is not formed on the end face of the weld defect shown by the synthesized 3D data due to the difference in protruding height of the peak of the weld defect shown by the first and second synthesis 3D data. Therefore, it is possible to prevent the prediction accuracy of the model from deteriorating due to the formation of the step on the end face of the weld defect shown by the synthesized 3D data.

[0011] According to the present disclosure, the effort and work time required to acquire learning data can be reduced.

[0012] FIG. 1 is a block diagram showing the configuration of a welding system including an AI model generation device as a learning data acquisition device according to an embodiment of the present disclosure. FIG. 2 is an explanatory diagram illustrating a 3D image represented by defect-free image data. FIG. 3 is a flowchart illustrating the operation of creating a defect detection model by the AI ​​model generation device. FIG. 4 is a flowchart illustrating the data expansion process by a data expansion unit. FIG. 5 is a plan view illustrating a set of 3D images represented by first and second selected 3D data and a 3D image represented by final 3D data. FIG. 6 is a perspective view of the set of 3D images shown in FIG. 5. FIG. 7 illustrates two 3D images represented by final 3D data. The 3D image on the left shows a 3D image of a welding defect in which an unintended step has been formed, and the 3D image on the right shows a 3D image of a welding defect in which an unintended step has not been formed.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. The following description of the preferred embodiments is merely exemplary in nature and is not intended to limit the present invention, its applications, or uses in any way.

[0014] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0015] 1 shows a welding system 1. The welding system 1 includes an AI model generation device 2 as a learning data acquisition device according to an embodiment of the present disclosure, and a welding device 3 that performs welding.

[0016] The AI ​​model generation device 2 creates a defect detection model through machine learning (deep learning). The AI ​​model generation device 2 also acquires learning data to be used in the machine learning for creating the defect detection model.

[0017] The defect detection model is an object detection model that predicts and outputs welding defect information based on input 3D data. The defect detection model is represented by a CNN (Convolutional Neural Network). The welding defect information includes category information indicating the type of welding defect in the 3D image represented by the 3D data, and position information indicating the position of the welding defect. In detail, the category information indicates whether the welding defect is a hole, pit, spatter, undercut, protrusion, etc. A pit is an opening on the surface of a weld bead as a weld mark. The position information indicates the position of a bounding box containing the welding defect. Note that the bounding box is a rectangular boundary that surrounds an object such as a welding defect.

[0018] The learning data generated by the AI ​​model generation device 2 is a plurality of data sets of 3D data representing 3D images and welding defect information.

[0019] Specifically, the AI ​​model generation device 2 includes a memory device 21, a processor 22 as a learning data acquisition unit, a first output device 23, and a first input device 24.

[0020] The storage device 21 has a defect image data storage unit 211, a defect-free image data storage unit 212, an annotation data storage unit 213, a segmentation data storage unit 214, a defect 3D data storage unit 215, a learning data storage unit 216, and a parameter storage unit 217.

[0021] The defect image data storage unit 211 stores, as defect image data, image data of a 3D image acquired by capturing an image of the area around the welding point PW including the welding defect using the 3D sensor 36 described later.

[0022] The defect-free image data storage unit 212 stores the defect-free image data generated by the processor 22 .

[0023] The annotation data storage unit 213 stores annotation data for the defect-containing image data stored in the defect-containing image data storage unit 211. This annotation data is acquired by a rectangular annotation. This annotation data specifies a rectangular area (bounding box) including a welding defect in the defect-containing image based on the defect-containing image data. The annotation data is specified by the user inputting defect information to the first input device 24 while the defect-containing image is being output to the first output device 23.

[0024] The segmentation data storage unit 214 stores segmentation data of the defect-containing image data stored in the defect-containing image data storage unit 211. This segmentation data specifies the boundary between a region where a welding defect has occurred and a region where a welding defect has not occurred in the defect-containing image based on the defect-containing image data. This segmentation data is specified by a boundary specification input by the user to the first input device 24 while the defect-containing image is being output to the first output device 23.

[0025] The segmentation data storage unit 214 also stores segmentation data of the defect-free image data stored in the defect-free image data storage unit 212. This segmentation data is acquired, for example, by polygon segmentation (polygon annotation). This segmentation data specifies the boundary between areas where welding defects are present and areas where they are not present in the defect-free image based on the defect-free image data. This segmentation data is created by the defect-free data processing unit 222.

[0026] The defective 3D data storage unit 215 stores a plurality of defective 3D data created by the processor 22 .

[0027] The learning data storage unit 216 stores the learning 3D data generated by the processor 22 .

[0028] The parameter storage unit 217 stores parameters that specify the fault detection model generated by the processor 22 .

[0029] The processor 22 has a defect-containing data processing unit 221, a defect-free data processing unit 222, a data extension unit 223, a learning data creation unit 224, and an AI model generation unit 225.

[0030] The defect data processing unit 221 receives defect image data from the welding device 3, which indicates a 3D image obtained by photographing the area around the welding point PW, including the welding defect, using the 3D sensor 36 described later, and stores it in the defect image data memory unit 211.

[0031] Furthermore, the defect-containing data processing unit 221 outputs a defect-containing image based on the defect-containing image data stored in the defect-containing image data storage unit 211 to the first output device 23. In this state, the first input device 24 accepts a user's input of defect information specifying the type and position of a welding defect in the defect-containing image. When the user has finished inputting the defect information, the defect-containing data processing unit 221 stores annotation data identified based on the user's input of the defect information to the first input device 24 in association with the defect-containing image data of each image in the annotation data storage unit 213.

[0032] Furthermore, the defect-containing data processing unit 221 outputs a defect-containing image based on the defect-containing image data stored in the defect-containing image data storage unit 211 to the first output device 23. In this state, the first input device 24 receives a boundary identification input from the user that identifies the boundary between the formed area and the non-formed area of ​​the welding defect in the defect-containing image. In response to the boundary identification input, the defect-containing data processing unit 221 stores segmentation data identified based on the user's boundary identification input to the first input device 24 in the segmentation data storage unit 214 in association with the defect-containing image data of each image.

[0033] The defect-free data processing unit 222 receives a base material designation input from the user to the first input device 24, which designates the size of the base material on which the weld bead is to be formed, and a weld mark identification input from the user to the first input device 24, which designates the shape of the weld bead. The defect-free data processing unit 222 then generates defect-free image data representing a defect-free image, which is a 3D image including the weld bead and the base material W, in accordance with the base material designation input and the weld mark identification input, and stores the defect-free image data in the defect-free image data storage unit 212. Figure 2 illustrates an example of a 3D image represented by the defect-free image data. The defect-free data processing unit 222 also automatically generates segmentation data identifying the boundary between the weld bead formation area and the weld bead non-formation area, and stores the segmentation data in the segmentation data storage unit 214.

[0034] The data extension unit 223 acquires a plurality of captured 3D data representing 3D images showing defective welds, based on the defect-containing image data stored in the defect-containing image data storage unit 211 and the annotation data of the defect-containing image data stored in the annotation data storage unit 213. The images represented by the captured 3D data are, for example, image D11 (corresponding to first selected 3D data (before compositing) described later) and image D21 (corresponding to second selected 3D data (before compositing) described later) shown in Figures 5 and 6, which show holes as defective welds.

[0035] The data extension unit 223 then performs data extension on the acquired plurality of pieces of captured 3D data to acquire a larger number of pieces of defective 3D data than the captured 3D data, and stores the defective 3D data in the defective 3D data storage unit 215 .

[0036] The learning data creation unit 224 creates learning image data based on the defect 3D data stored in the defect 3D data storage unit 215 and the defect-free image data stored in the defect-free image data storage unit 212. Specifically, the learning data creation unit 224 creates multiple learning 3D data by combining multiple patterns of one or more of the multiple welding defects represented by the multiple defect 3D data, and synthesizing the combined patterns with a 3D image represented by the defect-free image data. The image represented by each learning 3D data is an image in which one or more of the multiple welding defects represented by the multiple defect 3D data have been pasted onto the 3D image represented by the defect-free image data. The learning data creation unit 224 then stores learning data, which is made up of multiple data sets of the created learning 3D data and annotation data for the learning 3D data, in the learning data storage unit 216.

[0037] The AI ​​model generation unit 225 creates a fault detection model by performing machine learning using the training data stored in the training data storage unit 216. Specifically, the AI ​​model generation unit 225 specifies the weights and biases of each node constituting a CNN (convolutional neural network) that expresses the fault detection model as parameters that specify the fault detection model. Then, the AI ​​model generation unit 225 stores the specified parameters in the parameter storage unit 217.

[0038] The first output device 23 outputs a defect-containing image based on the defect-containing image data stored in the defect-containing image data storage unit 211. The first output device 23 is configured, for example, by a liquid crystal monitor. The first output device 23 also outputs an image prompting the user to perform various inputs.

[0039] The first input device 24 receives the defect information input, the boundary identification input, the base material designation input, and the weld mark identification input from the user, and transmits a signal corresponding to the received input to the processor 22.

[0040] The welding device 3 has a welding torch 31, a wire feeder (not shown), a welding power source 32, an output control unit 33, a robot arm 34, a robot control unit 35, a 3D sensor 36, a computer 37, a second output device 38, and a second input device 39. When power is supplied from the welding power source 32 to the welding wire WI held by the welding torch 31, an arc is generated between the tip of the welding wire WI and the base metal W, and the base metal W is heated to perform arc welding. Note that the welding device 3 has other components and equipment such as piping and gas cylinders for supplying shielding gas to the welding torch 31, but for convenience of explanation, these are not shown or described.

[0041] Output control unit 33 is connected to welding power source 32 and a wire feeder (not shown) and controls the welding output of welding torch 31, in other words, the power supplied to welding wire WI and the power supply time, in accordance with predetermined welding conditions. Output control unit 33 also controls the feed speed and feed amount of welding wire WI fed from the wire feeder (not shown) to welding torch 31. Note that the welding conditions may be input directly to output control unit 33 via an input unit (not shown), or may be selected from a welding program separately read from a recording medium or the like.

[0042] Robot arm 34 is a known articulated robot that holds welding torch 31 at its tip and is connected to robot control unit 35. Robot control unit 35 controls the operation of robot arm 34 so that the tip of welding torch 31, in other words, the tip of welding wire WI held by welding torch 31, moves to a desired position while tracing a predetermined welding trajectory.

[0043] The 3D sensor 36 is attached to the welding torch 31 and measures the shape of the welded portion PW of the base metal W, specifically, the area around the weld bead. The 3D sensor 36 is a three-dimensional shape measurement sensor that includes, for example, a laser light source (not shown) configured to scan the surface of the base metal W and a camera (not shown) that captures the reflection trajectory of the laser light projected onto the surface of the base metal W (hereinafter, sometimes referred to as a shape line). The 3D sensor 36 scans the entire welded portion PW of the base metal W with a laser beam, and the camera captures the laser beam reflected by the welded portion PW, thereby measuring the shape of the welded portion PW. The 3D sensor 36 is configured to measure the shape of not only the welded portion PW but also a predetermined area around it. This is to evaluate the presence or absence of spatter, etc. The camera has a CCD or CMOS image sensor as an imaging element. The configuration of the 3D sensor 36 is not limited to the above, and other configurations are possible. For example, an optical interferometer may be used instead of the camera.

[0044] The computer 37 executes software implemented on a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) to realize the functions of a plurality of functional blocks within the computer 37. The computer 37 has an image processing unit 371, a welding defect information calculation unit 372, and a welding defect display image generation unit 373.

[0045] The image processing unit 371 receives the shape data acquired by the 3D sensor 36 and, based on this shape data, acquires measurement 3D data of a 3D image including the welding point PW. Specifically, the image processing unit 371 acquires point cloud data of the shape lines captured by the 3D sensor 36. The image processing unit 371 also corrects the inclination and distortion of the base portion of the welding point PW relative to a predetermined reference plane, for example, the installation surface of the base material W, by statistically processing the point cloud data, and acquires measurement 3D data including the welding point PW.

[0046] The defective welding information calculation unit 372 calculates (acquires) defective welding information based on the measurement 3D data generated by the image processing unit 371, using the defect detection model generated by the AI ​​model generation device 2. The calculation of the defective welding information is executed when a predetermined input operation is performed on the second input device 39.

[0047] The defective welding display image generating unit 373 calculates a pixel value of each pixel of the defective welding display image displaying the defective welding information, based on the defective welding information calculated by the defective welding information calculating unit 372. The information specifying the defective welding display image may be compressed using a JPEG format or the like.

[0048] The second output device 38 displays the poor welding display image based on the pixel value of each pixel calculated by the poor welding display image generating unit 373. The second output device 38 is configured by a liquid crystal monitor or the like.

[0049] The second input device 39 accepts a predetermined input operation by the user to cause the computer 37 to start calculating the defective welding information.

[0050] Next, the operation of creating a fault detection model by the AI ​​model generation device 2 will be described with reference to the flowchart of FIG.

[0051] First, in (S101), the user starts the power supply to the welding power source 32 of the welding device 3 and performs arc welding to form a weld bead along with a weld defect on the base material W. Then, the user causes the 3D sensor 36 to capture an image of the welded portion PW of the base material W. Thereafter, the image processing unit 371 acquires defect image data that indicates an image including a weld defect based on the shape data acquired by the 3D sensor 36.

[0052] Next, in (S102), the defect data processing unit 221 of the AI ​​model generation device 2 receives the defect image data acquired in (S101) from the welding device 3, converts the data format from CSV data to NPY data, and stores it in the defect image data storage unit 211.

[0053] Next, in (S103), the defect-containing data processing unit 221 outputs a defect-containing image based on the defect-containing image data stored in the defect-containing image data storage unit 211 to the first output device 23. In this state, the first input device 24 accepts defect information input by the user, which specifies the type and position of a welding defect in the defect-containing image. When the user has finished inputting the defect information, the defect-containing data processing unit 221 associates annotation data identified based on the user's defect information input to the first input device 24 with the defect-containing image data of each image and stores the annotation data in the annotation data storage unit 213.

[0054] Next, in (S104), the defect-containing data processing unit 221 outputs a defect-containing image based on the defect-containing image data stored in the defect-containing image data storage unit 211 to the first output device 23. In this state, the defect-containing data processing unit 221 receives a boundary identification input from the user that identifies the boundary between the defective weld formation area and the non-defective area in the defect-containing image. In response to the boundary identification input, the defect-containing data processing unit 221 associates segmentation data indicating the boundary between the defective weld formation area and the non-defective area with the defect-containing image data of each image and stores it in the segmentation data storage unit 214.

[0055] Next, in (S105), the defect-free data processing unit 222 appropriately displays an image prompting the user to input information and accepts the base material designation input and the weld mark identification input. Note that an input specifying the position of the weld bead on the base material may also be included as the weld mark identification input. The defect-free data processing unit 222 then creates defect-free image data representing a 3D image including the weld bead and the base material. In the 3D image represented by the defect-free image data, for example, as shown in FIG. 2 , a weld bead having a shape specified by the weld mark identification input is formed on a base material having a size specified by the base material designation input.

[0056] Then, in (S106), the defect-free data processing unit 222 automatically creates segmentation data that identifies the boundary between the formation area and non-formation area of ​​the welding mark based on the defect-free image data, and stores the data in the segmentation data storage unit 214.

[0057] Next, in (S107), the data expansion unit 223 acquires a plurality of captured 3D data representing 3D images showing defective welds, based on the defect-containing image data stored in the defect-containing image data storage unit 211 and the annotation data of the defect-containing image data stored in the annotation data storage unit 213. Furthermore, the data expansion unit 223 acquires, as segmentation data of the captured 3D data, segmentation data that specifies the boundary between the formation area and non-formation area of ​​the defective weld in the 3D image represented by each captured 3D data, based on the segmentation data of the defect-containing image data stored in the segmentation data storage unit 214.

[0058] Next, in (S108), the data extension unit 223 repeatedly performs data extension, which will be described later, on the multiple pieces of captured 3D data acquired in (S107) to acquire a larger number of pieces of defective 3D data than the captured 3D data, and stores the defective 3D data in the defective 3D data storage unit 215. The detailed process of data extension will be described later.

[0059] Thereafter, in (S109), the learning data creation unit 224 creates learning image data based on the defect 3D data stored in the defect 3D data storage unit 215 and the defect-free image data stored in the defect-free image data storage unit 212. Specifically, the learning data creation unit 224 creates multiple learning 3D data by combining multiple patterns of one or more of the multiple welding defects represented by the multiple defect 3D data stored in the defect 3D data storage unit 215 and synthesizing the combined patterns with the 3D image represented by the defect-free image data. Then, the learning data creation unit 224 stores the created learning data, which is made up of multiple data sets of the learning 3D data and annotation data for the learning 3D data, in the learning data storage unit 216.

[0060] Next, in (S110), the AI ​​model generation unit 225 creates a defect detection model by performing machine learning using the learning data stored in the learning data storage unit 216. Then, the AI ​​model generation unit 225 stores parameters that specify the generated defect detection model in the parameter storage unit 217. The parameters stored in the parameter storage unit 217 are sent to the defective welding information calculation unit 372 of the welding device 3. The defective welding information calculation unit 372 stores the parameters.

[0061] The user then starts the power supply to the welding power source 32 of the welding device 3 and performs arc welding to form a weld bead on the base material W along with a weld defect. The 3D sensor 36 then captures an image of the welded portion PW, and the image processing unit 371 receives the shape data acquired by the 3D sensor 36. The image processing unit 371 then converts this shape data into image data of a welding image including the welded portion PW. The poor welding information calculation unit 372 then calculates (predicts) poor welding information based on the image data generated by the image processing unit 371 using a defect detection model identified by parameters sent from the AI ​​model generation device 2. The poor welding display image generation unit 373 calculates the pixel value of each pixel of the poor welding display image displaying the poor welding information based on the poor welding information calculated by the poor welding information calculation unit 372. The second output device 38 then displays the poor welding display image based on the pixel value of each pixel calculated by the poor welding display image generation unit 373.

[0062] Here, the detailed process of data extension performed by the data extension unit 223 in (S108) will be described with reference to the flowchart of FIG.

[0063] First, in (S201), the data expansion unit 223 randomly selects first and second selected 3D data from the multiple captured 3D data acquired in (S107). The first and second selected 3D data are data representing point clouds including a welding defect that is recessed or protruded in the Z-axis direction in a predetermined XYZ orthogonal coordinate system. FIGS. 5 and 6 illustrate pairs of the first and second selected 3D data and the final 3D data. FIGS. 5 and 6 show examples of 3D images of a hole as a welding defect represented by the 3D data. Images D11 and D12 are 3D images before synthesis represented by the first selected 3D data, and image D21 is a 3D image before synthesis represented by the second selected 3D data. Images D11, D12, and D21 in FIG. 5 are images of the welding defect viewed from the Z-axis direction. In FIGS. 5 and 6, the symbol B indicates the bottom of the hole. The data extension unit 223 uses the first selected 3D data as it is as the first 3D data to be combined (images D11 and D12). That is, the first 3D data to be combined is based on the first selected 3D data.

[0064] 5 and 6, image D12 is an image of the same sample as image D11, photographed under different lighting conditions. Images D22 to D28, which are obtained by performing multiple types of conversion processing on image D21, and images D31 to D38, which are obtained by combining the two images, will be described in detail later.

[0065] Next, in step S202, the data expansion unit 223 randomly selects one mode from among multiple modes that perform multiple types of conversion processes, including inversion of the welding defect in the X-axis direction and the Y-axis direction, transposition that swaps the X and Y values, and combinations thereof, and a mode that does not perform any of the multiple types of conversion processes. In the selected mode, the data expansion unit 223 then performs one of the conversion processes on the second selected 3D data (pre-conversion 3D data) representing, for example, image D21, and outputs the result as pre-size-adjusted 3D data (post-conversion 3D data), or outputs the second selected 3D data as pre-size-adjusted 3D data without performing the conversion process.

[0066] The multiple types of conversion processes include horizontal flipping (flipping in the X-axis direction), vertical flipping (flipping in the Y-axis direction), horizontal / vertical flipping (flipping in the X-axis and Y-axis directions), transposition in which the X and Y values ​​are swapped, horizontal flipping and transposition, vertical flipping and transposition, and horizontal / vertical flipping and transposition. In the examples of Figures 5 and 6, image D21 (corresponding to the second selected 3D data (before compositing)) is converted into an image of pre-size-adjusted 3D data. Specifically, image D21 is horizontally flipped to image D22, vertically flipped to image D23, horizontally and vertically flipped to image D24, transposed to image D25, horizontally flipped and transposed to image D26, vertically flipped to image D27, and horizontally and vertically flipped and transposed to image D28.

[0067] Next, in (S203), the data expansion unit 223 acquires second compositing 3D data (not shown) by enlarging or reducing the pre-size-adjusted 3D data (e.g., images D21-D28) in at least one of the X-axis and Y-axis directions so that the size of the pre-size-adjusted 3D data matches the size of the first selected 3D data (e.g., image D11 or image D12). As a result, the image sizes in the X-axis and Y-axis directions of the 3D images represented by the first compositing 3D data (e.g., images D11, D12) and the second compositing 3D data are mutually consistent. If the sizes in the X-axis and Y-axis directions of the pre-adjusted 3D data are the same as the sizes of the first selected 3D data (e.g., images D11, D12), the pre-size-adjusted 3D data is used as the second compositing 3D data as is. In other words, the second compositing 3D data is based on the converted 3D data and the second selected 3D data.

[0068] Next, in step S204, the data extension unit 223 normalizes the first and second compositing 3D data by multiplying the Z value of each point in the point cloud by a common value 1 / α to set the highest Z value of the point cloud to 1 and the lowest Z value to 0, thereby acquiring first and second normalized 3D data. In the present embodiment, in step S204, the data extension unit 223 normalizes the highest Z value to 1 and the lowest Z value to 0. However, normalization may be performed such that the highest Z value is set to a common first value other than 1 and the lowest Z value is set to a common second value other than 0. Normalization makes it possible to align the height or depth of the weld defects before compositing. Without normalization, the amount of change in the height difference of the weld defects before compositing significantly affects the shape of the weld defects after compositing, such as the formation of unintended steps in the weld defects after compositing.

[0069] Next, in (S205), the data extension unit 223 determines whether the welding defect in the first 3D data to be combined is concave (e.g., a hole) or convex (e.g., a spatter or protrusion) based on the first 3D data to be combined and the segmentation data of the first 3D data to be combined (acquired in (S107)). If the average Z value of the welding defect region in the 3D image represented by the first 3D data to be combined is equal to or less than the average Z value of the welding defect region, the data extension unit 223 determines that the welding defect is concave and proceeds to processing (S206). On the other hand, if the average Z value of the welding defect region in the 3D image represented by the first 3D data to be combined exceeds the average Z value of the welding defect region, the data extension unit 223 determines that the welding defect is convex and proceeds to processing (S207). That is, the data extension unit 223 selects which of steps (S206) and (S207) to execute based on the first compositing 3D data and the segmentation data of the first compositing 3D data (acquired in (S107)). Note that in this embodiment, the selection of which of steps (S206) and (S207) to execute is based on the first compositing 3D data and its segmentation data, but the selection may also be based on the second compositing 3D data, or the first and second compositing 3D data and their segmentation data.

[0070] In (S206), the data extension unit 223 generates the synthesized 3D data by selecting the value (smallest Z value) indicating the deeper Z value of each point among the Z values ​​of points whose XY coordinates are common to the point represented by the first and second normalized 3D data. That is, in the 3D image represented by the synthesized 3D data, the Z value of each point is set to the smaller of the Z values ​​of points whose XY coordinates are common to the point represented by the first and second normalized 3D data. For example, if the XYZ coordinates of a point included in the point cloud represented by the first normalized 3D data are (3, 3, -4) and the XYZ coordinates of a point included in the point cloud represented by the second normalized 3D data are (3, 3, -2), the XYZ coordinates (3, 3, -4) of the point included in the point cloud represented by the synthesized 3D data can be determined based on these coordinates. Therefore, in the 3D image represented by the combined 3D data, the bottom of the weld defect is formed at a certain depth in the region on the XY coordinate plane where the bottom of the weld defect is located in the first combined 3D data and in the region on the XY coordinate plane where the bottom of the weld defect is located in the second combined 3D data. After generating the combined 3D data, the data expansion unit 223 proceeds to processing (S208). Here, the bottom of the weld defect in the 3D data is, in other words, formed by 3D data representing a point cloud including a weld defect that is recessed in the Z-axis direction in a predetermined XYZ Cartesian coordinate system.

[0071] In step S207, the data expansion unit 223 generates composite 3D data by selecting the highest Z value (the maximum Z value) of the Z values ​​of points whose XY coordinates are the same as those of the point in the first and second normalized 3D data. In other words, in the 3D image represented by the composite 3D data, the Z value of each point is set to the higher Z value of the Z values ​​of points whose XY coordinates are the same as those of the point in the first and second normalized 3D data. Therefore, in the 3D image represented by the composite 3D data, the top of the weld defect is formed with a certain protruding height in the area on the XY coordinate plane where the top of the weld defect is located in the first 3D data for synthesis and the area on the XY coordinate plane where the top of the weld defect is located in the second 3D data for synthesis. After generating the composite 3D data, the data expansion unit 223 proceeds to step S208. Here, the peak of the welding defect in the 3D data is, in other words, formed by 3D data that indicates a point cloud including the welding defect that protrudes in the Z-axis direction in a predetermined XYZ orthogonal coordinate system.

[0072] In (S208), the data extension unit 223 performs a process on the composited 3D data by multiplying the Z coordinate of each point by a value α (the reciprocal of the value 1 / α multiplied during normalization in (S204)), thereby obtaining final 3D data (corresponding, for example, to images D31 to D38 in FIGS. 5 and 6). In other words, the final 3D data is based on the composited 3D data. The data extension unit 223 then stores the final 3D data in the defective 3D data storage unit 215.

[0073] The data expansion unit 223 repeatedly executes the processes (S201) to (S208) to obtain a large number of final 3D data. For example, in the examples of FIGS. 5 and 6, final 3D data representing images D31 to D34 can be obtained from the first and second selected 3D data representing images D11 and D21. Image D31 is an image obtained by combining images D11 and D21, image D32 is an image obtained by combining images D11 and D22, image D33 is an image obtained by combining images D11 and D23, and image D34 is an image obtained by combining images D11 and D24. Furthermore, final 3D data representing images D35 to D38 can be obtained from the first and second selected 3D data representing images D12 and D21. Image D35 is an image obtained by combining image D12 and image D25, image D36 is an image obtained by combining image D12 and image D26, image D37 is an image obtained by combining image D12 and image D27, and image D38 is an image obtained by combining image D12 and image D28. By setting the number of captured 3D data obtained in (S107) to three or more, a larger amount of more diverse final 3D data can be obtained compared to the case where only two data are obtained.

[0074] In this way, the number of 3D images of welding defects used to create 3D learning data can be increased by acquiring final 3D data based on the first and second selected 3D data without increasing the number of measurements by the 3D sensor 36. This reduces the effort and work time required to acquire learning data.

[0075] Furthermore, when the data expansion unit 223 executes step (S206), a bottom of a certain depth is formed in the final 3D data in the region on the XY coordinate plane where the bottom of the weld defect is located in the first compositing 3D data and in the region on the XY coordinate plane where the bottom of the weld defect is located in the second compositing 3D data. If the data expansion unit 223 does not execute steps (S204) to (S208), a step ST of an unrealistic shape may be generated, as shown in the left diagram of FIG. 7 . In contrast, in this embodiment, by executing steps (S204) to (S208), final 3D data showing a weld defect with a natural shape, as shown in the right diagram of FIG. 7 , can be generated. That is, a step is not formed on the bottom surface of the weld defect shown in the final 3D data due to the difference in the depth of the bottom of the weld defect shown in the first and second compositing 3D data. Therefore, deterioration of the prediction accuracy of the defect detection model caused by the formation of the step on the bottom surface of the weld defect shown in the final 3D data can be suppressed.

[0076] Furthermore, when the data expansion unit 223 executes step (S207), a peak of a certain protruding height is formed in the final 3D data in the region on the XY coordinate plane where the peak of the weld defect is located in the first compositing 3D data and in the region on the XY coordinate plane where the peak of the weld defect is located in the second compositing 3D data. If the data expansion unit 223 does not execute steps (S204) to (S208), a step of a shape that does not actually occur may be generated. In contrast, in this embodiment, by executing steps (S204) to (S208), final 3D data showing a weld defect with a natural shape can be generated. That is, a step is not formed on the end face of the weld defect shown in the final 3D data due to the difference in the protruding height of the weld defect's peak shown in the first and second compositing 3D data. Therefore, it is possible to prevent the prediction accuracy of the defect detection model from being degraded due to the formation of the step on the end face of the weld defect shown in the final 3D data.

[0077] In the above embodiment, the combined 3D data is obtained by combining two pieces of combined 3D data. However, the combined 3D data may be obtained by combining three or more pieces of combined 3D data. In this case, first, each of the three or more pieces of combined 3D data is normalized to normalized 3D data. Then, if the welding defect is concave, the combined 3D data can be generated by selecting the minimum Z value of each point among the Z values ​​of points that share the same XY coordinates as the point shown in the three or more normalized 3D data. On the other hand, if the welding defect is convex, the combined 3D data can be generated by selecting the maximum Z value of each point among the Z values ​​of points that share the same XY coordinates as the point shown in the three or more normalized 3D data.

[0078] In the above embodiment, the welding device 3 used to form the weld bead to obtain image data with defects in (S101) and the welding device 3 used to form the weld bead to obtain image data to be input into the defect detection model are a common welding device, but they may also be different welding devices.

[0079] In the above embodiment, the functions of the defect-containing data processing unit 221, the defect-free data processing unit 222, the data extension unit 223, and the learning data creation unit 224, and the function of the AI ​​model generation unit 225 are realized by one processor 22, but they may be realized by different processors. In other words, the functions of the processor 22 in the above embodiment may be realized by multiple processors.

[0080] The training data acquisition device and training data acquisition method disclosed herein can reduce the effort and work time required to acquire training data, and are useful as a training data acquisition device and training data acquisition method for acquiring image data to be used in machine learning.

[0081] 2 AI model generation device (learning data acquisition device) 22 Processor (learning data acquisition unit) D11, D12, D21 to D28, D31 to D38 3D image

Claims

1. A learning data acquisition device for acquiring learning data used in machine learning for creating a model, wherein the model outputs welding defect information including the position of a welding defect based on 3D data indicating a 3D image, and the learning data acquisition device includes: a first synthesis step of generating synthesized 3D data showing a 3D image in which the bottom of the welding defect is formed at a certain depth in a region on the XY coordinate plane where the bottom of the welding defect is located in the first synthesis 3D data and a region on the XY coordinate plane where the bottom of the welding defect is located in the second synthesis 3D data, based on first and second synthesis 3D data showing a point group including a welding defect recessed in the Z-axis direction in a predetermined XYZ orthogonal coordinate system and having common image sizes in the X-axis and Y-axis directions; and at least one of a second synthesis step of generating synthesized 3D data showing a 3D image in which the top of the welding defect is formed at a certain protrusion height in a region on the XY coordinate plane where the top of the welding defect is located in the first synthesis 3D data and a region on the XY coordinate plane where the top of the welding defect is located in the second synthesis 3D data, based on first and second synthesis 3D data showing a point group including a welding defect protruding in the Z-axis direction in a predetermined XYZ orthogonal coordinate system and having common image sizes in the X-axis and Y-axis directions; and a learning data creation step of creating a plurality of 3D data for learning by combining 3D images of a plurality of welding defects shown by a plurality of defective 3D data including the final 3D data based on the synthesized 3D data in a plurality of patterns and synthesizing them with a 3D image including a weld bead. A learning data acquisition device characterized by comprising a learning data acquisition unit that executes the above steps.

2. In the learning data acquisition device according to claim 1, in the first synthesis step, for the first and second 3D data for synthesis, normalization is performed by multiplying the Z value of each point in the point cloud by a predetermined value 1 / α common to the Z values of the points in the point cloud, so that the maximum value of the Z value of the point cloud becomes a common first value and the minimum value becomes a common second value, thereby obtaining first and second normalized 3D data; and as the Z value of each point, a value indicating deeper among the Z values of points having the same XY coordinates as the point represented by the first and second normalized 3D data is selected, thereby generating the synthesized 3D data in a first Z value selection step. In the second synthesis step, the normalization step is performed, and as the Z value of each point, a value indicating higher among the Z values of points having the same XY coordinates as the point represented by the first and second normalized 3D data is selected, thereby generating the synthesized 3D data in a second Z value selection step. The learning data acquisition unit obtains the final 3D data by performing a process of multiplying the Z coordinate of each point in the synthesized 3D data by the reciprocal α of the predetermined value 1 / α. A learning data acquisition device characterized by the above.

3. In the learning data acquisition device according to claim 2, the learning data acquisition unit selects which of the first and second synthesis steps to execute based on at least one of the first and second 3D data for synthesis and segmentation data specifying a boundary between a formed region and a non-formed region of the welding defect in the 3D image represented by the at least one 3D data for synthesis. A learning data acquisition device characterized by the above.

4. In the learning data acquisition device according to claim 1, the learning data acquisition unit further performs a size adjustment step of obtaining the second 3D data for synthesis by enlarging or reducing the 3D data before size adjustment in at least one of the X-axis direction and the Y-axis direction. A learning data acquisition device characterized by the above.

5. In the learning data acquisition device according to claim 1, the learning data acquisition unit further executes a 3D data selection step of randomly selecting first and second selected 3D data from a plurality of 3D data indicating welding defects, and the first 3D data for synthesis is based on the first selected 3D data, and the second 3D data for synthesis is based on the second selected 3D data. A learning data acquisition device characterized by this.

6. In the learning data acquisition device according to claim 1, the learning data acquisition unit includes a mode selection step of randomly selecting one mode from a plurality of modes each performing a plurality of types of conversion processes including inversion, transposition, and combinations thereof of welding defects, and a mode in which none of the plurality of types of conversion processes are performed, and in the mode selected in the mode selection step, any one of the conversion processes is performed on the pre-conversion 3D data indicating welding defects to output the post-conversion 3D data, or the pre-conversion 3D data is directly output as the post-conversion 3D data without performing the conversion process. Further execute a conversion process step, and the second 3D data for synthesis is based on the post-conversion 3D data. A learning data acquisition device characterized by this.

7. A learning data acquisition method for acquiring learning data used in machine learning for creating a model, wherein the model outputs welding defect information including the position of a welding defect based on 3D data indicating a 3D image, and the learning data acquisition method includes: showing a point group including a welding defect that is concave in the Z-axis direction in a predetermined XYZ orthogonal coordinate system, and based on first and second 3D data for synthesis having the same image sizes in the X-axis direction and the Y-axis direction, an area on the XY coordinate plane where the bottom of the welding defect is located in the first 3D data for synthesis, and an area on the XY coordinate plane where the bottom of the welding defect is located in the second 3D data for synthesis, a first synthesis step of generating synthesized 3D data showing a 3D image in which the bottom of the welding defect is formed at a certain depth, and showing a point group including a welding defect that protrudes in the Z-axis direction in a predetermined XYZ orthogonal coordinate system, and based on first and second 3D data for synthesis having the same image sizes in the X-axis direction and the Y-axis direction, an area on the XY coordinate plane where the top of the welding defect is located in the first 3D data for synthesis, and an area on the XY coordinate plane where the top of the welding defect is located in the second 3D data for synthesis, at least one of the first synthesis step and a second synthesis step of generating synthesized 3D data showing a 3D image in which the top of the welding defect is formed at a certain protruding height; and a learning data creation step of creating a plurality of 3D data for learning by combining 3D images of a plurality of welding defects shown by a plurality of defective 3D data including the final 3D data based on the synthesized 3D data with a 3D image including a weld bead. The first synthesis step includes: a normalization step of performing normalization on the first and second 3D data for synthesis by multiplying the Z value of each point in the point group by a predetermined value 1 / α common to the Z values of each point to make the maximum value of the Z values of the point group a common first value and the minimum value a common second value, thereby obtaining first and second normalized 3D data; and a first Z value selection step of generating the synthesized 3D data by selecting, as the Z value of each point, a value indicating a deeper one among the Z values of points having the same XY coordinates as the point indicated by the first and second normalized 3D data. The second synthesis step includes: the normalization step,As the Z value of each point, a value indicating a higher one among the Z values of the points having the same XY coordinates as the point indicated by the first and second normalized 3D data is selected, and a second Z value selection step of generating the synthesized 3D data is executed. The learning data acquisition method is characterized in that the final 3D data is acquired by performing a process of multiplying the Z coordinate of each point of the synthesized 3D data by the reciprocal α of the predetermined value 1 / α.

8. In the learning data acquisition method according to claim 7, a mode selection step of randomly selecting one mode from a plurality of modes each performing a plurality of types of conversion processes including inversion, transposition, and combinations thereof of welding defects, and a mode in which none of the plurality of types of conversion processes are performed, and in the mode selected in the mode selection step, any one of the conversion processes is performed on the pre-conversion 3D data indicating welding defects to output the post-conversion 3D data, or the pre-conversion 3D data is directly output as the post-conversion 3D data without performing the conversion process. Further execute a conversion process step, and the second 3D data for synthesis is based on the post-conversion 3D data. A learning data acquisition method characterized by this.

Images