Learning model generation method, trained model, image processing method, image processing system, and welding system

The learning model generation method enhances feature extraction accuracy in welding systems by training on multiple images with varying feature positions, improving positional accuracy and welding precision.

JP7594403B2Active Publication Date: 2024-12-04KK TOSHIBA
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2020167698
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-10-02
Publication Date
2024-12-04
Estimated Expiration
2040-10-02

AI Technical Summary

Technical Problem

Existing image processing techniques for welding systems lack high feature extraction accuracy, particularly in estimating features from input images using trained learning models.

Method used

A learning model generation method that includes training data with multiple input images and feature extraction images, utilizing a teacher data approach to train a learning model with a convolutional input layer where feature positions differ, and incorporating changes in feature positions across images to enhance accuracy.

Benefits of technology

The method enables high-accuracy feature extraction from multiple input images, improving positional accuracy and welding precision by leveraging changes in feature positions across images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007594403000001
    Figure 0007594403000001
  • Figure 0007594403000002
    Figure 0007594403000002
  • Figure 0007594403000003
    Figure 0007594403000003
Patent Text Reader

Abstract

To provide a method for generating a learning model, a learned model, an image processing method, an image processing system, and a welding system, with high extraction accuracy of a feature.SOLUTION: A method for generating a learning model comprises the steps of: acquiring teacher data including a plurality of learning input images, and a learning feature extraction image obtained by extracting a feature from one of the plurality of learning input images; and training a learning model outputting an extraction image of the feature estimated from a plurality of input images using the teacher data. The learning model includes an input layer that performs a convolution. Positions of the feature in each of the plurality of learning input images are different from each other. A change amount of the position of the feature in the plurality of learning input images is less than a kernel size of a filter of the input layer.SELECTED DRAWING: Figure 10
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The embodiments relate to a method for generating a learning model, a trained model, an image processing method, an image processing system, and a welding system. [Background technology]

[0002] Conventionally, there has been known a technique for estimating an image having extracted features from an input image using a trained learning model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-141902 A Summary of the Invention [Problem to be solved by the invention]

[0004] The embodiments aim to provide a method for generating a learning model with high feature extraction accuracy, a learned model, an image processing method, an image processing system, and a welding system. [Means for solving the problem]

[0005] A method for generating a learning model according to an embodiment includes the steps of acquiring training data including a plurality of training input images and a training feature-extracted image in which features are extracted from one of the plurality of training input images, and training a learning model using the training data to output an extracted image of the features estimated from the plurality of input images. The learning model includes an input layer that performs convolution. The positions of the features in each of the plurality of training input images are different from each other. The amount of change in the position of the feature in the plurality of training input images is smaller than a kernel size of a filter in the input layer. [Brief description of the drawings]

[0006] [Figure 1] 1 is a diagram showing a welding system according to a first embodiment. [Diagram 2] FIG. 2(a) is a top view showing the welded members before welding, and FIG. 2(b) is a top view showing the welded members during welding. [Diagram 3] 1 is a block diagram showing a hardware configuration of a control device in a welding system according to a first embodiment. [Figure 4] FIG. 2 is a diagram showing a learning model according to the first embodiment. [Diagram 5] 4 is a flowchart showing a method for generating a learning model according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing data used for learning a learning model according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing a method of preprocessing an input image for learning, which is one of the methods for generating a learning model according to the first embodiment. [Figure 8] FIG. 2 is a diagram illustrating a generator of a learning model according to the first embodiment. [Figure 9] FIG. 9(a) is a diagram showing the processing of the input layer in the learning model according to the first embodiment, and FIG. 9(b) is a diagram showing a convolution method in the input layer. [Figure 10] 13 is a diagram showing that the positions of the contours of the molten pool are different from one another in a plurality of learning input images. FIG. [Figure 11] FIG. 11(a) is a diagram showing the processing of the first hidden layer in the learning model related to the first embodiment, FIG. 11(b) is a diagram showing the processing of the second hidden layer in the learning model related to the first embodiment, and FIG. 11(c) is a diagram showing the processing of the third hidden layer in the learning model related to the first embodiment. [Figure 12] Figure 12(a) is a diagram showing the processing of the fourth hidden layer of the learning model related to the first embodiment, Figure 12(b) is a diagram showing the processing of the fifth hidden layer of the learning model related to the first embodiment, and Figure 12(c) is a diagram showing the processing of the sixth hidden layer of the learning model related to the first embodiment. [Figure 13]FIG. 2 is a diagram showing the processing of an output layer in a learning model according to the first embodiment. [Figure 14] FIG. 2 is a diagram showing a feature extraction image output by a learning model according to the first embodiment. [Figure 15] 4 is a flowchart showing a welding method using a learning model according to the first embodiment. [Figure 16] FIG. 4 is a diagram showing a portion of a welding system according to a second embodiment. [Figure 17] FIG. 13 is a diagram showing a portion of a welding system according to a third embodiment. [Figure 18] FIG. 13 is a diagram showing a portion of a welding system according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] <First embodiment> First, the first embodiment will be described. FIG. 1 is a diagram showing a welding system according to the present embodiment. FIG. 2(a) is a top view showing the welded members before welding, and FIG. 2(b) is a top view showing the welded members during welding.

[0008] (Welding Systems) A welding system 10 according to this embodiment welds two or more welded members together. The welding system 10 performs, for example, laser welding or arc welding. Here, an example will be described in which the welding system 10 mainly performs laser welding of two welded members 21, 22 as shown in Figs. 2(a) and 2(b). Hereinafter, the two welded members 21, 22 will also be referred to as a "first welded member 21" and a "second welded member 22".

[0009] The first welded member 21 and the second welded member 22 are, for example, plate-shaped members. The first welded member 21 and the second welded member 22 are arranged to face each other. Hereinafter, the surface of the first welded member 21 facing the second welded member 22 will be referred to as the "first surface 21a," and the surface of the second welded member 22 facing the first welded member 21 will be referred to as the "second surface 22a."

[0010] As shown in FIG. 1, welding system 10 includes, for example, welding section 11, imaging device 15, lighting device 16, and control device 17.

[0011] In the following, an XYZ Cartesian coordinate system is used for ease of understanding. The direction from first welded member 21 and second welded member 22 toward head 13 is defined as the "Z direction." The direction perpendicular to the Z direction, from first welded member 21 toward second welded member 22, is defined as the "Y direction." The direction perpendicular to the Z direction, that is, the traveling direction of head 13, is defined as the "X direction."

[0012] The welding section 11 includes a light source 12, a head 13, and an arm 14. The head 13 is connected to the light source 12, and irradiates the laser light L emitted by the light source 12 onto a first member to be welded 21 and a second member to be welded 22. The arm 14 holds the head 13, and moves the head 13 relative to the first member to be welded 21 and the second member to be welded 22. The arm 14 is capable of moving the head 13, for example, in the X direction, the Y direction, and the Z direction.

[0013] The photographing device 15 is, for example, a camera including a CCD image sensor or a CMOS image sensor. The photographing device 15 is disposed above the first welded member 21 and the second welded member 22. In this embodiment, the photographing device 15 photographs a moving image D of the welding point during welding. Hereinafter, the moving image D is also referred to as a "control moving image D".

[0014] The lighting device 16 illuminates the welding area so that a clearer image can be obtained by the imaging device 15. If an image that can be used for image processing by the image processing system described later can be obtained without illuminating the welding area, the lighting device 16 does not have to be provided.

[0015] FIG. 3 is a block diagram showing the hardware configuration of a control device in the welding system according to this embodiment. In this embodiment, the control device 17 is a computer including a GPU (Graphics Processing Unit) 17a, a ROM (Read Only Memory) 17b, a RAM (Random Access Memory) 17c, a hard disk 17d, etc. The GPU 17a, the ROM 17b, the RAM 17c, and the hard disk 17d are connected to each other via a bus 17e. However, the configuration of the control device is not limited to the above. For example, the control device may use other processors such as a CPU instead of a GPU. The control device may also include other configurations such as an input / output interface.

[0016] 1, in this embodiment, the control device 17 functions as an acquisition unit 171, an image processing unit 172, a control unit 173, and a storage unit 174. The functions of the acquisition unit 171, the image processing unit 172, and the control unit 173 are realized by, for example, a GPU 17a. Also, the function of the storage unit 174 is realized by, for example, a ROM 17b, a RAM 17c, a hard disk 17d, etc.

[0017] When welding first member 21 and second member 22 to be welded, control unit 173 controls welding unit 11 to move head 13 in the X direction while emitting laser light L from head 13 toward first member 21 and second member 22 to be welded. Control unit 173 also controls imaging device 15 to capture a video D of the welding point during welding.

[0018] As shown in FIG. 2(b), the first welded member 21 and the second welded member 22 are irradiated with the laser light L, and a part of the first welded member 21 and a part of the second welded member 22 are melted to generate a molten pool 31. In the X direction, which is the moving direction of the head 13, the unmelted first surface 21a and the second surface 22a are present in front of the molten pool 31. In addition, in a region in the molten pool 31 where the energy density of the irradiated laser light L is high, the molten metal may evaporate and a keyhole 32 may be generated. Then, the molten pool 31 solidifies, and the first welded member 21 and the second welded member 22 are integrated. A weld bead 33 is formed at the joint between the first welded member 21 and the second welded member 22. Therefore, each image constituting the video D includes any one of the first surface 21a, the second surface 22a, the molten pool 31, the keyhole 32, and the weld bead 33.

[0019] The acquisition unit 171 acquires a plurality of images from among the images constituting the video D as a plurality of control input images IA1, IA2, and IA3 at a predetermined time interval during welding as shown in FIG. 1. Here, an example in which the number of control input images is three will be described, but the number of control input images is not particularly limited as long as it is two or more. For example, the control input image IA3 is the latest image, and the control input image IA2 is an image taken at a time immediately before the control input image IA3. The control input image IA1 is an image taken at a time immediately before the control input image IA2. However, the control input image IA2 does not have to be an image taken immediately before the control input image IA3, and the control input image IA1 does not have to be an image taken at a time immediately before the control input image IA2.

[0020] The image processing unit 172 uses the trained learning model 200 stored in the storage unit 174 to output a feature extraction image IB estimated from a plurality of control input images IA1, IA2, and IA3 at a predetermined time interval during welding. Hereinafter, the feature extraction image IB is also referred to as a "control feature extraction image IB." The trained learning model 200 is also referred to as a "trained model 200."

[0021] The features extracted by the image processing unit 172 are contours of specific regions in a plurality of control input images IA1, IA2, and IA3, etc. Here, an example in which the image processing unit 172 extracts a plurality of features will be described. However, the number of features extracted by the image processing unit is not particularly limited as long as it is 1 or more.

[0022] The image processor 172 extracts the contour of the molten pool 31 as a line R1, the contour of the keyhole 32 as a line R2, the first surface 21a which is a part of the contour of the first welded member 21 as a line R3, and the second surface 22a which is a part of the contour of the second welded member 22 as a line R4 from the multiple control input images IA1, IA2, and IA3. That is, the control feature extraction image IB is an image in which the contour of the molten pool 31 is shown as a line R1, the contour of the keyhole 32 is shown as a line R2, the first surface 21a is shown as a line R3, and the second surface 22a is shown as a line R4. However, the features extracted by the image processor are not particularly limited to those described above. For example, the image processor may extract the contour of a weld bead as a feature.

[0023] The control unit 173 controls the welding portion 11 at a predetermined time interval using the control feature extraction image IB. Specifically, the control unit 173 calculates the deviation between the center position of the keyhole 32 in the Y direction from the control feature extraction image IB and the center position of the gap between the first surface 21a and the second surface 22a in front of the keyhole 32 in the Y direction, and controls the arm 14 to eliminate the deviation. The control unit 173 also controls the output of the light source 12 so that the position of the outline of the molten pool 31 in the control feature extraction image IB in the Y direction is located outside the first surface 21a and the second surface 22a and falls within a certain range. This can improve the positional accuracy of the welding between the first welded member 21 and the second welded member 22 and the strength of the welding.

[0024] (Learning model) Next, the trained model 200 used in the welding system 10 will be described. FIG. 4 is a diagram showing a learning model according to the present embodiment. The learning model 200 used in the welding system 10 has been trained using the training data TD.

[0025] The teacher data TD includes a plurality of learning input images IC1, IC2, and IC3, and a feature-extracted learning image ID2 in which features are extracted from one of the plurality of learning input images IC1, IC2, and IC3. The number of learning input images IC1, IC2, and IC3 used by the learning model 200 in one learning is the same as the number of control input images IA1, IA2, and IA3 used in one image processing, and is, for example, three.

[0026] The multiple learning input images IC1, IC2, and IC3 are, for example, three images among images constituting a learning video of a welding point. The learning video is captured by, for example, a photographing device 15. For example, the learning input image IC1 is an image captured at a time immediately before the learning input image IC2, and the learning input image IC3 is an image captured immediately after the learning input image IC2. However, the photographing device for the learning video and the photographing device for the control video D may be different.

[0027] The learning feature extraction image ID2 is, for example, an image in which features are extracted from the learning input image IC2, and is prepared by a user of the generating device 40 described later before learning the learning model 200. Specifically, the learning feature extraction image ID2 is an image in which the contour of the molten pool 31 in the learning input image IC2 is shown as a line R5, the contour of the keyhole 32 is shown as a line R6, the first surface 21a is shown as a line R7, and the second surface 22a is shown as a line R8, similar to the control feature extraction image IB. The learning feature extraction image ID2 is created, for example, by tracing the parts in the learning input image IC2 that the creator has identified as the contour of the molten pool 31, the contour of the keyhole 32, the first surface 21a, and the second surface 22a with lines, and extracting the traced lines. However, the method of creating the learning feature extraction image is not limited to the above. The learning feature extraction image may also be, for example, an image in which features are extracted from the learning input image IC1 or the learning input image IC3.

[0028] The algorithm used in the learning model 200 is an algorithm for generating an image from an image, such as pix2pix.

[0029] The learning model 200 includes a generator 210 and a discriminator 220. The generator 210 outputs an extracted image IE of features estimated from a plurality of learning input images IC1, IC2, and IC3. When a pair of a learning input image IC2 and a learning feature extracted image ID2 and a pair of a learning input image IC2 and a feature extracted image IE generated by the generator 210 are input, the discriminator 220 discriminates which pair is the teacher data TD, i.e., the real one, and which pair is not the teacher data TD, i.e., the fake one. The learning of the generator 210 is advanced so that the discriminator 220 discriminates the pair of the learning input image IC2 and the feature extracted image IE generated by the generator 210 as the real one. Furthermore, the learning of the classifier 220 is advanced so that the pair of the learning input image IC2 and the learning feature extracted image ID2 can be identified as genuine, and the pair of the learning input image IC2 and the feature extracted image IE generated by the generator 210 can be identified as fake. Specific processes performed by the generator 210 and the classifier 220 will be described later.

[0030] In this embodiment, the learning model 200 is generated by a generating device 40 as shown in Fig. 1. The generating device 40 is a computer including a processor such as a GPU or a CPU, a ROM, a RAM, a hard disk, etc. However, the control device 17 may generate the learning model.

[0031] (How to generate a learning model) Next, a method for generating the learning model 200 will be described. FIG. 5 is a flowchart showing a method for generating a learning model according to this embodiment. The method for generating the learning model 200 includes a step S11 of acquiring training data TD, a step S12 of preprocessing the learning input images IC1, IC2, and IC3, and a step S13 of training the learning model 200. Each step will be described in detail below.

[0032] FIG. 6 is a diagram showing data used for learning of the learning model according to this embodiment. First, the generating device 40 acquires the teacher data TD prepared in advance by the user (step S11). That is, the generating device 40 acquires a plurality of learning input images IC1, IC2, and IC3, and a learning feature-extracted image ID2 in which features are extracted from the learning input image IC2.

[0033] In this embodiment, the generating device 40 further obtains a pre-processing feature extraction image ID1 in which features are extracted from the learning input image IC1, and a pre-processing feature extraction image ID3 in which features are extracted from the learning input image IC3. In the pre-processing feature extraction images ID1 and ID3, similar to the learning feature extraction image ID2, the contour of the molten pool 31 is extracted as a line R5, the contour of the keyhole 32 is extracted as a line R6, the first surface 21a is extracted as a line R7, and the second surface 22a is extracted as a line R8, and these images are prepared in advance by the user. Similarly to the learning feature extraction image ID2, the pre-processing feature extraction images ID1 and ID3 are generated by the creator tracing the parts recognized as the contour of the molten pool 31, the contour of the keyhole 32, the first surface 21a, and the second surface 22a in the learning input images IC1 and IC3 with lines and extracting the traced lines.

[0034] FIG. 7 is a diagram showing a method of preprocessing an input image for learning, which is part of the method of generating a learning model according to this embodiment. Next, the generating device 40 preprocesses the learning input images IC1, IC2, and IC3 (step S12).

[0035] Specifically, the generating device 40 uses the pre-processing feature extraction image ID1 to create a first mask M1 in which the values ​​of the pixels constituting the lines R5, R6, R7, R8 and the periphery of the lines R5, R6, R7, R8 are set to zero, and the other pixels are set to one. Hereinafter, the image is also considered as a matrix, and the pixels are also referred to as "elements." The generating device 40 also creates a second mask M2 in which the values ​​of the elements constituting the lines R5, R6, R7, R8 and the periphery of the lines R5, R6, R7, R8 in the pre-processing feature extraction image ID1 are set to one, and the other elements are set to one. In FIG. 7, for ease of explanation, elements in the first mask M1 and the second mask M2 with a value of zero are shown in black, and elements with a value of one are shown in white.

[0036] Next, the generating device 40 multiplies the elements of the learning input image IC1 and the first mask M1 together. Here, "multiplying elements together" means that in two matrices, such as the learning input image IC1 and the first mask M1, the process of multiplying the element in the i-th row and j-th column of one matrix by the element in the i-th row and j-th column of the other matrix is ​​performed for all elements. This creates an image M4 from the learning input image IC1 with the features and their surroundings removed.

[0037] The generating device 40 also applies a filter such as a smoothing filter, a Gaussian filter, or a median filter to the learning input image IC1 to create an image M3 in which the learning input image IC1 is entirely blurred. The term "blur" refers to a process of reducing changes in gradation in an image. The generating device 40 then multiplies the entirely blurred image M3 by elements of the second mask M2. This creates an image M5 in which the features and the areas surrounding the features are extracted from the entirely blurred image M3.

[0038] Next, the generating device 40 generates the training input image IC1 and the first mask M1The elements of image M4, which is obtained by multiplying image M3, which has been entirely blurred, and image M5, which is obtained by multiplying image M3 with the first mask M2, are added together. Here, "adding elements together" means that in two matrices, the element in the i-th row and j-th column of one matrix is ​​added to the element in the i-th row and j-th column of the other matrix, and this process is carried out for all elements. This creates a preprocessed image IM1.

[0039] By performing the above-mentioned processing, a preprocessed image IM1 can be obtained in which the features and surroundings of the training input image IC1 are blurred and other areas are not blurred. The generating device 40 performs the same processing on the training input image IC2 to generate a preprocessed image IM2 of the training input image IC2. The generating device 40 also performs the same processing on the training input image IC3 to generate a preprocessed image IM3 of the training input image IC3.

[0040] In step S12, the degree of blurring of the multiple learning input images IC1, IC2, IC3 may be the same or different from each other. The degree of blurring of each learning input image IC1, IC2, IC3 can be adjusted by a weighting value when applying a filter such as a smoothing filter, a Gaussian filter, or a median filter. When the degree of blurring of the multiple preprocessed images IM1, IM2, IM3 is different from each other, the learning model 200 is trained so that features can be extracted from the preprocessed image with the maximum degree of blurring among the multiple preprocessed images IM1, IM2, IM3.

[0041] However, an image obtained by blurring the entire learning input image may be used as a preprocessed image and input to the input layer of the learning model described later. Also, learning input images that are not preprocessed may be input to the input layer.

[0042] FIG. 8 is a diagram showing a generator of a learning model according to this embodiment. Next, the generating device 40 trains the learning model 200 using the multiple preprocessed images IM1, IM2, and IM3 and the feature-extracted learning image ID2 (step S13).

[0043] In this embodiment, U-NET is used for the generator 210. Specifically, in this embodiment, the generator 210 includes an input layer 211, a first hidden layer 212a, a second hidden layer 212b, a third hidden layer 212c, a fourth hidden layer 213a, a fifth hidden layer 213b, a sixth hidden layer 213c, and an output layer 214. Note that, although an example in which the number of hidden layers 212a, 212b, 212c, 213a, 213b, and 213c is six is ​​shown in FIG. 8, the number of hidden layers is not limited to the above.

[0044] FIG. 9(a) is a diagram showing the processing of the input layer in the learning model according to this embodiment, and FIG. 9(b) is a diagram showing a convolution method in the input layer. In the following, for ease of explanation, in a matrix of an image, filter, etc., the direction in which elements are arranged in a row is referred to as the "horizontal direction x," and the direction in which elements are arranged in a column is referred to as the "vertical direction y."

[0045] A plurality of preprocessed images IM1, IM2, and IM3 are input as a set of data to the input layer 211. In the input layer 211, the set of preprocessed images IM1, IM2, and IM3 is convoluted. In the following, an example will be described in which convolution is performed with b filters F11, F12 to F1b in the input layer 211, and the kernel size of each of the filters F11 to F1b is n1×n1.

[0046] First, the generating device 40 extracts an area A1 of the same size as the filter F11 in the preprocessed image IM1. Next, the generating device 40 calculates a value r1(i,j) by multiplying an element im1(i,j) in the i-th row and j-th column of the extracted area A1 by an element f1(i,j) in the i-th row and j-th column of the filter F11. The generating device 40 performs the same process for all elements im1(i,j) in the area A1. Next, the generating device 40 calculates a value c1(p,q) by adding up all the values ​​r1(i,j) calculated for the area A1.

[0047] Similarly, the generating device 40 extracts an area A2 in the preprocessed image IM2 that is the same size as the filter F11 and is located at the same position as the area A1. Next, the generating device 40 calculates a value r2(i,j) by multiplying an element im2(i,j) in the i-th row and j-th column of the extracted area A2 by an element f1(i,j) in the i-th row and j-th column of the filter F11. The generating device 40 performs the same process for all elements im2(i,j) in the area A2. Next, the generating device 40 calculates a value c2(p,q) by adding up all the values ​​r2(i,j) calculated for the area A2.

[0048] Similarly, the generating device 40 extracts an area A3 in the preprocessed image IM3 that is the same size as the filter F11 and is located at the same position as the area A1. Next, the generating device 40 calculates a value r3(i,j) by multiplying an element im3(i,j) in the i-th row and j-th column of the extracted area A3 by an element f1(i,j) in the i-th row and j-th column of the filter F11. The generating device 40 performs the same process for all elements im3(i,j) in the area A3. Next, the generating device 40 calculates a value c3(p,q) by adding up all the values ​​r3(i,j) calculated for the area A3.

[0049] Next, the generating device 40 calculates a value cs(p, q) by adding together the calculated values ​​c1(p, q), c2(p, q), and c3(p, q).

[0050] Next, the generating device 40 sequentially shifts the regions A1, A2, A3 to which the filter F11 is applied for each of the preprocessed images IM1, IM2, IM3 in the horizontal direction x, and calculates the value cs(p,q) in the same manner. After shifting the regions A1, A2, A3 to the last row of each of the preprocessed images IM1, IM2, IM3, the generating device 40 shifts the regions A1, A2, A3 back to the first row and shifts each of the regions A1, A2, A3 in the vertical direction y, and performs the same processing. The above processing is repeated until each of the regions A1, A2, A3 is shifted onto an element belonging to the last row and the last column of each of the preprocessed images IM1, IM2, IM3.

[0051] In this embodiment, in the input layer 211, each of the regions A1, A2, and A3 is shifted by one element in the horizontal direction x or the vertical direction y. That is, the stride is 1. When each of the regions A1, A2, and A3 is shifted, if each of the regions A1, A2, and A3 protrudes from the preprocessed images IM1, IM2, and IM3, zero padding is performed to set the values ​​of the elements of the protruding parts of each of the regions A1, A2, and A3 to zero. However, each of the regions A1, A2, and A3 may be shifted by two or more elements at a time. That is, the stride may be two or more.

[0052] 9(a), a first feature map P11 is created in which the element in the pth row and qth column has the value cs(p, q). As described above, in this embodiment, the regions A1, A2, and A3 are shifted by one element each in the horizontal direction x and vertical direction y. Therefore, the size of the first feature map P11 is the same as the size of each of the preprocessed images IM1, IM2, and IM3.

[0053] Next, the same process as that for the filter F11 is performed on the filters F12 to F1b. As a result, a plurality of first feature maps P12 to P1b are created. In this manner, in the input layer 211, the three preprocessed images IM1, IM2, and IM3 are convoluted as a set of data.

[0054] FIG. 10 is a diagram showing that the positions of the contours of the molten pool are different from one another in a plurality of learning input images. In FIG. 10, the position of the contour of the molten pool 31 in the learning input image IC1 is indicated by a line R5a. IC2 The position of the contour of the molten pool 31 in the learning input image IC3 is indicated by line R5b, and the position of the contour of the molten pool 31 in the learning input image IC3 is indicated by line R5c. The multiple learning input images IC1, IC2, and IC3 have different feature positions, and the changes Δx and Δy in the feature positions of the multiple learning input images IC1, IC2, and IC3 are smaller than the kernel size n1 of each of the filters F11 to F1b.

[0055] For example, when the laser beam L is continuously irradiated onto a certain region of the first workpiece 21 and the second workpiece 22, the molten pool 31 gradually expands. In this case, when a video of the welding point is captured by the imaging device 15, the positions of the contours of the molten pool 31 differ from one another in the images constituting the video.

[0056] In this embodiment, a combination of images constituting the moving image in which the maximum change amount Δx in the horizontal direction x of the contour position of the molten pool 31 and the maximum change amount Δy in the vertical direction y of the contour position of the molten pool 31 are smaller than the kernel size n1 of each filter F11 to F1b is selected as each learning input image IC1, IC2, IC3. In order to perform such selection, the time interval at which the image capturing device 15 captures images, i.e., the frame rate, is set so that the change amounts Δx and Δy in the feature positions of the multiple learning input images IC1, IC2, IC3 are smaller than the kernel size n1 of each filter F11 to F1b. When the frame rate is determined, the kernel size n1 may be reduced so that the change amounts Δx and Δy in the feature positions of the multiple learning input images IC1, IC2, IC3 are smaller than the kernel size n1 of each filter F11 to F1b. Similarly, the angle of view may be increased.

[0057] For the other features, that is, the outline of the keyhole 32, and the first and second surfaces 21a and 22a, the learning input images IC1, IC2, and IC3 are selected so as to satisfy similar requirements.

[0058] By selecting a plurality of learning input images IC1, IC2, and IC3 as described above, for example, if a feature is included in an area A1 of the same size as the filter F11 in one learning input image IC1, the possibility that the feature is also included in areas A2 and A3 of the same size as the filter F11 in the other learning input images IC2 and IC3 increases. Therefore, the learning model 200 can learn to estimate an extracted image IE of a feature from the plurality of learning input images IC1, IC2, and IC3 by incorporating information on changes in the positions of the features in the plurality of learning input images IC1, IC2, and IC3. As a result, even if it is difficult to extract the position of a feature from a single image, the position of the feature can be captured and extracted with high accuracy from the changes in the position of the feature in the plurality of images. As a result, the feature extraction accuracy can be improved when a plurality of control input images IA1, IA2, and IA3 are input to the learning model 200.

[0059] In this embodiment, an example has been described in which the positions of the features of the multiple learning input images IC1, IC2, and IC3 are based on the passage of time. That is, in this embodiment, the amounts of change Δx and Δy occur due to the passage of time. However, as in other embodiments described later, the amounts of change do not have to occur due to the passage of time.

[0060] Figure 11(a) is a diagram showing the processing of the first hidden layer in the learning model of this embodiment, Figure 11(b) is a diagram showing the processing of the second hidden layer in the learning model of this embodiment, and Figure 11(c) is a diagram showing the processing of the third hidden layer in the learning model of this embodiment. Next, as shown in FIG. 11(a), the multiple first feature maps P11 to P1b created in the input layer 211 are input to the first hidden layer 212a.

[0061] In the first hidden layer 212a, a plurality of first feature maps P12 to P1b are convolved as a set of data by c filters F21, F22 to F2c. Note that the specific method of convolution is the same as the method of convolution in the input layer 211, except that in the image to be convolved, an area of ​​the same size as each of the filters F21 to F2c is shifted by two or more elements. Therefore, a detailed description of the convolution in the first hidden layer 212a will be omitted.

[0062] In the first hidden layer 212a, a plurality of second feature maps P21, P22-P2c are created by convoluting a plurality of first feature maps P12-P1b with c filters F21-F2c. In this embodiment, in each of the first feature maps P11-P1b, the region to which each of the filters F21-F2c is applied is shifted by two or more elements. Therefore, the size of the plurality of second feature maps P21-P2c is smaller than the size of the plurality of first feature maps P12-P1b.

[0063] Next, as shown in Fig. 11(b), in the second hidden layer 212b, the second feature maps P21 to P2c are convoluted as a set of data with d filters F31, F32 to F3d. As a result, d third feature maps P31, P32 to P3d are created. In this embodiment, in each of the second feature maps P21 to P2c, the region to which each of the filters F31 to F3d is applied is shifted by two or more elements. Therefore, the size of the third feature maps P31 to P3d becomes smaller than the size of the second feature maps P21 to P2c.

[0064] Next, as shown in Fig. 11(c), in the third hidden layer 212c, the third feature maps P31 to P3d are convoluted as a set of data with e filters F41, F42 to F4e. As a result, e fourth feature maps P41, P42 to P4e are created. In this embodiment, in each of the third feature maps P31 to P3d, the region to which each of the filters F41 to F4e is applied is shifted by two or more elements. Therefore, the size of the fourth feature maps P41 to P4e is smaller than the size of the third feature maps P31 to P3d.

[0065] Furthermore, in this embodiment, the amounts of change Δx, Δy in the feature positions of the multiple learning input images IC1, IC2, and IC3 are smaller than the kernel size n2 of each of the filters F21-F2c in the first intermediate layer 212a, the kernel size n3 of each of the filters F31-F3d in the second intermediate layer 212b, and the kernel size n4 of each of the filters F41-F4e in the third intermediate layer 212c. This makes it easier to propagate information regarding changes in the feature positions included in the multiple first feature maps P11-P1b from the first intermediate layer 212a to the third intermediate layer 212c.

[0066] Figure 12(a) is a diagram showing the processing of the fourth hidden layer of the learning model of this embodiment, Figure 12(b) is a diagram showing the processing of the fifth hidden layer of the learning model of this embodiment, and Figure 12(c) is a diagram showing the processing of the sixth hidden layer of the learning model of this embodiment. Next, the multiple fourth feature maps P41-P4e created by the third hidden layer 212c are input to the fourth hidden layer 213a. In the fourth hidden layer 213a, the multiple fourth feature maps P41-P4e are deconvoluted as a set of data. "Deconvolution" refers to a process in which, assuming that the input feature map was created by convolving a map with some kind of filter, the input feature map is convoluted with a filter equivalent to the transposed matrix of the filter.

[0067] Specifically, first, first enlarged maps K11, K12 to K1e are created by enlarging the size of each of the fourth feature maps P41 to P4e in the horizontal direction x and the vertical direction y. Each of the enlarged maps K11 to K1e is created by adding an element whose value is zero to each of the fourth feature maps P41 to P4e. Next, f filters F51, F52 to F5f are convoluted with a set of data of the first enlarged maps K11, K12, K13 to K1e. As a result, f fifth feature maps P51, P52 to P5f are created. Here, the f filters F51, F52 to F5f correspond to the transposed matrix of a filter when it is assumed that the fourth feature maps P41 to P4e are created by convoluting a map with a filter. This makes it possible to make the size of the output fifth feature maps P51 to P5f larger than the size of the input fourth feature maps P41 to P4e.

[0068] Next, as shown in FIG. 12(b), a plurality of fifth feature maps P51 to P54 created by the fourth intermediate layer 213a are P5f The third feature maps P31 to P3d created in the second hidden layer 212b are input to the fifth hidden layer 213b. P5f and the third feature maps P31 to P3d are deconvoluted as a set of data.

[0069] Specifically, in the fifth hidden layer 213b, second enlarged maps K21-K2f are created by enlarging the horizontal x size and vertical y size of the plurality of fifth feature maps P51-P5f, and third enlarged maps K31-K3d are created by enlarging the horizontal x size and vertical y size of the plurality of third feature maps P31-P3d. Next, the plurality of second enlarged maps K21-K2f and the third enlarged maps K31-K3d are convoluted as a set of data with g filters F61, F62-F6g. As a result, g sixth feature maps P61, P62-P6g are created. The size of the output plurality of sixth feature maps P61-P6g is larger than the size of the input plurality of fifth feature maps P51-P5f.

[0070] 12(c), the sixth feature maps P61-P6g created by the fifth hidden layer 213b and the second feature maps P21-P2c created by the first hidden layer 212a are input to the sixth hidden layer 213c. In the sixth hidden layer 213c, the sixth feature maps P61-P6g and the second feature maps P21-P2c are deconvoluted as a set of data.

[0071] Specifically, in the sixth hidden layer 213c, fourth enlarged maps K41-K4g are created by enlarging the horizontal x size and vertical y size of the sixth feature maps P61-P6g, and fifth enlarged maps K51-K5c are created by enlarging the horizontal x size and vertical y size of the second feature maps P21-P2c. Next, the fourth enlarged maps K41-K4g and the fifth enlarged maps K51-K5c are convoluted as a set of data with h filters F71, F72-F7h. As a result, h seventh feature maps P71, P72-P7h are created. The size of the output seventh feature maps P71-P7h is larger than the size of the input sixth feature maps P61-P6g.

[0072] In this embodiment, the amounts of change Δx, Δy in the feature positions of the multiple learning input images IC1, IC2, and IC3 are smaller than the kernel size n5 of each of the filters F51-F5f in the fourth hidden layer 213a, the kernel size n6 of each of the filters F61-F6g in the fifth hidden layer 213b, and the kernel size n7 of each of the filters F71-F7h in the sixth hidden layer 213c. This makes it easier to propagate information about changes in the feature positions included in the multiple fourth feature maps P41-P4e from the fourth hidden layer 213a to the sixth hidden layer 213c.

[0073] FIG. 13 is a diagram showing the processing of the output layer in the learning model according to this embodiment. 13, in the output layer 214, the seventh feature maps P71 to P7h ​​are convolved as a set of data with three filters F81, F82, and F83, thereby creating three eighth feature maps P81, P82, and P83.

[0074] In this embodiment, the amount of change in feature positions in the multiple learning input images IC1, IC2, and IC3 is smaller than the kernel size n8 of the filters F81-F83 in the output layer 214. Therefore, the learning model 200 can learn to estimate a feature extraction image IE from the multiple learning input images IC1, IC2, and IC3 by incorporating the changes in feature positions in the multiple learning input images IC1, IC2, and IC3.

[0075] In the learning model 200, for example, the number c of the filters F21 to F2c in the first intermediate layer 212a is greater than the number b of the filters F11 to F1b in the input layer 211. Also, the number d of the filters F31 to F3d in the second intermediate layer 212b is greater than the number c of the filters F21 to F2c in the first intermediate layer 212a. Also, the number e of the filters F41 to F4e in the third intermediate layer 212c is greater than the number d of the filters F31 to F3d in the second intermediate layer 212b. Also, the number f of the filters F51 to F5f in the fourth intermediate layer 213a is the same as the number e of the filters F41 to F4e in the third intermediate layer 212c. Also, the number g of the filters F61 to F6g in the fifth intermediate layer 213b is the same as the number d of the filters F31 to F3d in the second intermediate layer 212b. Furthermore, the number h of the filters F71 to F7h in the sixth intermediate layer 213c is the same as the number c of the filters F21 to F2c in the first intermediate layer 212a. However, the magnitude relationship between b to h is not limited to the above.

[0076] In the learning model 200, for example, the kernel size n1 of the input layer 211 is the same as the kernel size n8 of the output layer 214. For example, the kernel sizes n2 to n7 of the intermediate layers 212a, 212b, 212c, 213a, 213b, and 213c are the same and larger than the kernel size n1 of the input layer 211. However, the magnitude relationship of the kernel sizes n1 to n8 is not limited to the above.

[0077] FIG. 14 is a diagram showing a feature extraction image output by a generator of a learning model according to this embodiment. In the eighth feature map P81, a portion estimated to be the contour of the molten pool 31 is extracted as a line R9. In the eighth feature map P82, a portion estimated to be the contour of the keyhole 32 is extracted as a line R10. In the eighth feature map P83, a portion estimated to be the first surface 21a is extracted as a line R11, and a portion estimated to be the second surface 22a is extracted as a line R12. A combination of the three eighth feature maps P81, P82, and P83 corresponds to an extracted image IE of features estimated from a plurality of learning input images IC1, IC2, and IC3.

[0078] Next, the discriminator 220 is input with a pair of the learning input image IC2 and the learning feature extraction image ID2, and a pair of the learning input image IC2 and the feature extraction image IE output by the generator 210. The discriminator 220 then discriminates which pair is the real pair and which is the fake pair. The generator 210 trains the discriminator 220 to discriminate the pair of the learning input image IC2 and the feature extraction image IE output by the generator 210 as the real pair, and determines the values ​​of the filter elements when performing convolution and deconvolution. The discriminator 220 also trains to discriminate the pair of the learning input image IC2 and the learning feature extraction image ID2 as the real pair, and the pair of the learning input image IC2 and the feature extraction image IE output by the generator 210 as the fake pair. The learning of the generator 210 and the learning of the discriminator 220 are performed simultaneously, so that the learning of both advances.

[0079] (Welding method) Next, a welding method using the learning model 200 according to this embodiment will be described. FIG. 15 is a flowchart showing a welding method using the learning model according to this embodiment. In the following description, during welding, control unit 173 controls welding unit 11 to emit laser light L from head 13 and gradually move head 13 in the X direction. Also, during welding, control unit 173 controls imaging device 15 to capture video D of the welded area during welding.

[0080] When welding starts, first, the acquisition unit 171 acquires, among the images constituting the video D of the welding point captured by the image capturing device 15, the latest image and two images captured immediately before that as a plurality of control input images IA1, IA2, and IA3 (step S21). In this embodiment, the frame rate and the angle of view of the image capturing device 15 are set so that the amount of change in the positions of the features of the plurality of control input images IA1, IA2, and IA3 is smaller than the kernel size n1 of the filters F11 to F1b of the input layer 211.

[0081] Next, the image processing unit 172 outputs an extracted image IB of features estimated from the three input images IA1, IA2, and IA3 using the learning model 200 stored in the storage unit 174 (step S22).

[0082] Next, the control unit 173 controls the welding portion 11 based on the feature extraction image IB output by the image processing unit 172 (step S23). Specifically, the control unit 173 calculates the deviation between the center position of the keyhole 32 in the Y direction from the feature extraction image IB for control and the center position of the gap between the first surface 21a and the second surface 22a in front of the keyhole 32 in the Y direction, and controls the arm 14 to eliminate the deviation. The control unit 173 also controls the output of the light source 12 so that the position of the outline of the molten pool 31 in the feature extraction image IB for control is located outside the first surface 21a and the second surface 22a and falls within a certain range.

[0083] Next, the control unit 173 judges whether the welding is completed or not (step S24). If it is judged that the welding is completed (step S24: Yes), the control unit 173 turns off the laser output and ends the welding. If it is judged that the welding is not completed (step S24: No), the processing from steps S21 to S24 is performed again.

[0084] Next, the effects of this embodiment will be described. The method for generating the learning model 200 according to the present embodiment includes the steps of acquiring training data TD including a plurality of learning input images IC1, IC2, and IC3, and a learning feature-extracted image ID2 in which features are extracted from one of the plurality of learning input images IC1, IC2, and IC3, and training the learning model 200, which outputs an extracted image IB of features estimated from the plurality of input images IA1, IA2, and IA3, using the training data TD. The learning model 200 includes an input layer 211 that performs convolution. The positions of the features in the plurality of learning input images IC1, IC2, and IC3 are different from each other. The change amounts Δx and Δy of the positions of the features in the plurality of learning input images IC1, IC2, and IC3 are smaller than the kernel size n1 of the filters F11 to F1b in the input layer 211.

[0085] In such a method for generating the learning model 200, the learning model 200 can be trained to estimate a feature extraction image IE from a plurality of learning input images IC1, IC2, and IC3 from information incorporating changes in the positions of the features in the plurality of learning input images IC1, IC2, and IC3. Therefore, when a plurality of input images IA1, IA2, and IA3 are input, the learning model 200 can extract features with high accuracy.

[0086] The learning model 200 also includes an output layer 214 that performs convolution. The changes Δx and Δy are smaller than the kernel size n8 of the filters F81, F82, and F83 of the output layer 214. Therefore, the learning model 200 can be trained to estimate an extracted image IE of features from a plurality of learning input images IC1, IC2, and IC3 from information incorporating changes in the positions of features in the plurality of learning input images IC1, IC2, and IC3. Therefore, when a plurality of input images IA1, IA2, and IA3 are input, the learning model 200 can extract features with high accuracy.

[0087] The learning model 200 also includes intermediate layers 212a, 212b, and 212c that perform convolution. The change amounts Δx and Δy are smaller than the kernel sizes n2, n3, and n4 of the filters F21 to F2c, F31 to F3d, and F41 to F4e of the intermediate layers 212a, 212b, and 212c. Therefore, the learning model 200 can be trained to estimate an extracted image IE of features from a plurality of learning input images IC1, IC2, and IC3 from information that incorporates changes in the positions of features in the plurality of learning input images IC1, IC2, and IC3. Therefore, the learning model 200 can extract features with high accuracy when a plurality of input images IA1, IA2, and IA3 are input.

[0088] The learning model 200 also includes intermediate layers 213a, 213b, and 213c that perform deconvolution. The change amounts Δx and Δy are smaller than the kernel sizes n5, n6, and n7 of the filters F51 to F5f, F61 to F6g, and F7 to F7h of the intermediate layers 213a, 213b, and 213c. Therefore, the learning model 200 can be trained to estimate an extracted image IE of features from a plurality of learning input images IC1, IC2, and IC3 from information that incorporates changes in the positions of features in the plurality of learning input images IC1, IC2, and IC3. Therefore, the learning model 200 can extract features with high accuracy when a plurality of input images IA1, IA2, and IA3 are input.

[0089] In addition, the learning model uses U-NET. That is, the feature maps P21-2c and P31-3d output from the first hidden layer 212a and the second hidden layer 212b are input to the deconvolution layers such as the fifth hidden layer 213b and the sixth hidden layer 213c. Therefore, when a plurality of input images IA1, IA2, and IA3 are input, the learning model 200 can extract features with high positional accuracy.

[0090] Moreover, the method for generating the learning model 200 according to this embodiment further includes a step of creating preprocessed images in which the features of a plurality of learning input images IC1, IC2, and IC3 are blurred before the learning step. In the learning step, the preprocessed images IM1, IM2, and IM3 are input to the input layer 211. Therefore, the learning model 200 can be trained so that features can be extracted even under strict conditions in which the features are blurred.

[0091] In addition, in the process of creating a preprocessed image, the degree of blurring of features in one of the multiple learning input images IC1, IC2, and IC3 is different from the degree of blurring of features in other learning input images IC1, IC2, and IC3. Therefore, even when the degree of blurring of features differs, the learning model 200 can be trained to extract features.

[0092] In addition, the multiple learning input images IC1, IC2, and IC3 are images constituting a video of the welding point corresponding to the target location. Therefore, multiple learning input images IC1, IC2, and IC3 with different feature positions can be easily prepared.

[0093] Furthermore, one of the multiple learning input images IC1, IC2, and IC3 is an image captured immediately before or after the other learning input images, so that multiple learning input images IC1, IC2, and IC3 in which the change amounts Δx and Δy in the feature positions are smaller than the kernel size n1 of the filters F11 to F1b can be easily prepared.

[0094] Further, the multiple learning input images IC1, IC2, and IC3 are images taken of the welding point during welding, and the features are at least a part of the contour of the molten pool 31, at least a part of the contour of the keyhole 32, or at least a part of the contours of the workpieces 21 and 22. Therefore, the features related to welding can be extracted with high accuracy.

[0095] The trained model 200 according to the present embodiment includes an input layer 211 that performs convolution, and has been trained using teacher data TD including a plurality of training input images IC1, IC2, and IC3, and a training feature-extracted image ID2 in which features are extracted from one of the plurality of training input images IC1, IC2, and IC3. The positions of the features of the plurality of training input images IC1, IC2, and IC3 are different from each other, and the change amounts Δx and Δy of the positions of the features in the plurality of training input images IC1, IC2, and IC3 are smaller than the kernel size n1 of the filters F11 to F1b of the input layer 211. The trained model 200 causes a computer to output a feature-extracted image IB estimated from the plurality of input images IA1, IA2, and IA3. Therefore, when a plurality of input images IA1, IA2, and IA3 are input, a trained model 200 that can extract features with high accuracy can be provided.

[0096] Moreover, the image processing method according to this embodiment includes a step of acquiring a plurality of input images IA1, IA2, and IA3, and a step of outputting an extracted image IB of features estimated from the plurality of input images IA1, IA2, and IA3 using the trained model 200. Therefore, it is possible to provide an image processing method capable of extracting features with high accuracy when a plurality of input images IA1, IA2, and IA3 are input.

[0097] Furthermore, the amount of change in the positions of the features in the multiple input images IA1, IA2, and IA3 is smaller than the kernel size n1 of the filters F11 to F1b in the input layer 211. Therefore, when multiple input images IA1, IA2, and IA3 are input, features can be extracted with high accuracy.

[0098] The image processing system according to this embodiment also includes an image processing unit 172 that outputs an extracted image IB of welding features estimated from a plurality of input images IA1, IA2, and IA3 using the trained model 200. Therefore, it is possible to provide an image processing system that can extract features with high accuracy when a plurality of input images IA1, IA2, and IA3 are input.

[0099] Furthermore, welding system 10 according to this embodiment includes welding section 11 that welds multiple welded members 21, 22, photographing device 15 that photographs the welding points of multiple welded members 21, 22, image processing section 172 that outputs extracted image IB of welding features estimated from multiple images photographed by photographing device 15 using learning model 200, and control section 173 that controls the welding device based on feature extraction image IB output by image processing section 172. Therefore, it is possible to provide welding system 10 that can create feature extraction image IB based on multiple input images IA1, IA2, IA3 and control welding work with high accuracy.

[0100] <Second embodiment> Next, a second embodiment will be described. FIG. 16 is a diagram showing a part of the welding system according to this embodiment. In the following description, in principle, only the differences from the first embodiment will be described, and the matters other than those described below are the same as those in the first embodiment.

[0101] In the first embodiment, an example was described in which images constituting the video D captured by the image capture device 15 are used as the control input images IA1, IA2, and IA3 and the learning input images IC1, IC2, and IC3. In contrast, in the present embodiment, the welding system 310 includes the image capture device 315 capable of acquiring a plurality of images with different wavelengths, polarization, or exposure times. The positions of features may differ from one another in the plurality of images with different wavelengths, polarization, or exposure times. The plurality of images with different wavelengths, polarization, or exposure times captured by the image capture device 315 may be used as the control input images IA1, IA2, and IA3 and the learning input images IC1, IC2, and IC3.

[0102] The photographing device 315 may have built-in filters capable of transmitting light of mutually different wavelengths, and the photographing device 315 may acquire images corresponding to each filter. In this case, one lighting device 16 may emit multiple lights of mutually different wavelengths, or may emit light of a wide band including multiple lights of mutually different wavelengths, or multiple lighting devices 16 may be provided, and the multiple lighting devices 16 may emit light of mutually different wavelengths. Also, the photographing device 315 may have built-in polarizers capable of transmitting light of mutually different polarization directions, and the photographing device 315 may acquire images corresponding to each polarizer. Also, the photographing device 315 may acquire a non-polarized image and a polarized image. In these cases, one lighting device 16 may emit multiple lights of mutually different polarization directions, or multiple lighting devices 16 may be provided, and the multiple lighting devices 16 may emit light of mutually different polarization directions. Also, the photographing device 315 may have built-in shutters capable of acquiring images with mutually different exposure times, and the photographing device 315 may acquire images corresponding to each exposure time.

[0103] In such a case, the wavelengths or the number of imaging devices are adjusted so that the amount of change in the feature positions of the learning input images IC1, IC2, and IC3 is smaller than the kernel size n1 of the filters F11 to F1b of the input layer 211. polarization is set.

[0104] <Third embodiment> Next, a third embodiment will be described. FIG. 17 is a diagram showing a part of the welding system according to this embodiment. In this embodiment, the welding system 410 includes a plurality of image capturing devices 415a, 415b, and 415c, which capture images of the welding point from different positions. The images captured by the plurality of image capturing devices 415a, 415b, and 415c may be used as input images IA1, IA2, and IA3 for control and input images IC1, IC2, and IC3 for learning.

[0105] In such a case, the positions of the multiple image capturing devices 415a, 415b, 415c are adjusted so that the amount of change in the feature positions of the learning input images IC1, IC2, IC3 is smaller than the kernel size n1 of the multiple filters F11 to F1b of the input layer 211.

[0106] <Fourth embodiment> Next, a fourth embodiment will be described. FIG. 18 is a diagram showing a part of the welding system according to this embodiment. In this embodiment, the welding system 510 includes a plurality of image capturing devices 515a, 515b, and 515c, which have different image capturing angles. Images captured by the plurality of image capturing devices 515a, 515b, and 515c may be used as input images IA1, IA2, and IA3 for control and input images IC1, IC2, and IC3 for learning.

[0107] In such a case, the shooting angles of the multiple image capturing devices 515a, 515b, and 515c are adjusted so that the amount of change in the feature positions of the learning input images IC1, IC2, and IC3 is smaller than the kernel size n1 of the multiple filters F11 to F1b of the input layer 211.

[0108] As described above, the multiple learning input images are images in which the shooting conditions when the welding points were photographed are different from each other. The shooting conditions are not particularly limited, but as described above, include the time when the welding points were photographed, the polarization direction of light, the shooting position, the shooting angle, the wavelength of light, and the exposure time. Similarly, the multiple control input images are images in which the shooting conditions when the welding points were photographed are different from each other. Note that, in the above embodiment, a form in which one shooting condition is different has been described, but multiple shooting conditions may be different.

[0109] In the above embodiment, the imaging device images the welded area during welding, but the imaging device may image the welded area after welding. When the imaging device images the welded area after welding, the image processing system may extract, for example, a weld bead as a feature, and the feature-extracted image output by the image processing system may be used for determining the accuracy of welding, etc.

[0110] In the above embodiment, the image processing system is realized by the control device of the welding system. However, the device that realizes the image processing system is not limited to the above. The image processing system may be realized by an edge device attached to the imaging device. The image processing system may be realized by a computer that processes images uploaded to the cloud. The image processing system may be realized by multiple computers.

[0111] The image processing system may also be applied to systems other than welding systems.

[0112] Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0113] 10, 310, 410, 510: Welding systems 11: Welded parts 12:Light source 13: Head 14: Arm 15, 315, 415a, 415b, 415c, 515a, 515b, 515c: Imaging device 16: Lighting equipment 17: Control device 17b :ROM 17c:RAM 17d: Hard disk 17e: Bus 21, 22: Welded parts 21a: 1st page 22a: 2nd side 31: Molten pool 32: Keyhole 33: Weld bead 40:Generation device 171: Acquisition Department 172: Image processing section 173: Control section 174: Storage section 200: Learning model 210: Generator 211: Input layer 212a: 1st middle layer 212b: 2nd middle layer 212c: 3rd intermediate layer 213a: 4th intermediate layer 213b: 5th middle layer 213c: 6th middle layer 214: Output layer 220: Classifier A1, A2, A3: Area F11~F1b: Filter F21~F2c: Filter F31~F3d: Filter F41~F4e: Filter F51~F5f: Filter F61~F6g: Filters F71~F7h: Filter F81~F83: Filter IA1, IA2, IA3: Input images for multiple controls IB: Feature extraction image for control IC1, IC2, IC3: Multiple training input images ID2: Feature extraction image for learning ID1, ID3: Feature extraction images for preprocessing IE: Feature extraction images for learning IM1~IM3: Preprocessed images K11~K1e: First enlarged map K21~K2f: 2nd enlarged map K31~K3d: 3rd expanded map K41~K4g: 4th expanded map K51~K5c: 5th expanded map L: Laser light M1: First mask M2: Second mask M3: Image with the whole image blurred P11~P1b: First feature map P21~P2c: Second feature map P31~P3d: Third feature map P41~P4e: 4th feature map P51~P5e: 5th feature map P61~P6g: 6th feature map P71~P7h: 7th feature map P81~P83: 8th feature map R1, R2~R12, R5a, R5b, R5c: line TD: Teacher data f1 : element im1 : element im2 : element im3 : element n1~n8: Kernel size x : horizontal y : vertical direction Δx: Amount of change Δy: Amount of change

Claims

1. acquiring training data including a plurality of learning input images and a feature-extracted learning image in which features are extracted from one of the plurality of learning input images; A step of training a learning model that outputs an extracted image of the features estimated from a plurality of input images using the training data; Equipped with The learning model includes an input layer that performs convolution; the positions of the features in the plurality of learning input images are different from one another; A method for generating a learning model, wherein an amount of change in the position of the feature in the multiple learning input images is smaller than a kernel size of a filter in the input layer.

2. The learning model includes an output layer that performs convolution, The method for generating a learning model according to claim 1 , wherein the amount of change is smaller than a kernel size of a filter of the output layer.

3. The learning model includes an intermediate layer that performs convolution; The method for generating a learning model according to claim 1 or 2, wherein the amount of change is smaller than a kernel size of a filter in the intermediate layer.

4. The learning model includes another intermediate layer that performs deconvolution, The method for generating a learning model according to claim 3 , wherein the amount of change is smaller than a kernel size of a filter of the other intermediate layer.

5. The method for generating a learning model according to any one of claims 1 to 4, wherein U-NET is used for the learning model.

6. The method further comprises the step of creating a plurality of pre-processed images in which the features of the plurality of learning input images are blurred, prior to the learning step; The method for generating a learning model according to any one of claims 1 to 5, wherein in the learning step, the plurality of preprocessed images are input to the input layer.

7. In the step of generating a plurality of preprocessed images, 7. The method for generating a learning model according to claim 6, wherein a degree of blurring of the feature in one learning input image among the plurality of learning input images is different from a degree of blurring of the feature in another learning input image among the plurality of learning input images.

8. The method for generating a learning model according to any one of claims 1 to 7, wherein the plurality of learning input images are images taken under different shooting conditions when the target location was photographed.

9. The method for generating a learning model according to claim 8, wherein the plurality of learning input images are images that differ from each other in at least one of the shooting conditions when the target area was photographed, including time, polarization direction of light, shooting position, shooting angle, wavelength of light, and exposure time.

10. The method for generating a learning model according to any one of claims 1 to 7, wherein the plurality of learning input images are images constituting a video of a target location.

11. the plurality of learning input images are images of a welding point photographed during welding, A method for generating a learning model according to any one of claims 1 to 10, wherein the feature is at least a part of the contour of a molten pool, at least a part of the contour of a keyhole, or at least a part of the contour of a workpiece to be welded.

12. It includes an input layer that performs convolution, The method has been trained using training data including a plurality of training input images and a training feature-extracted image in which features are extracted from one of the plurality of training input images; the positions of the features in the plurality of learning input images are different from one another; a change in the position of the feature in the plurality of learning input images is smaller than a kernel size of a filter in the input layer; A trained model that causes a computer to output an extracted image of the features estimated from multiple input images.

13. acquiring a plurality of input images; outputting an extracted image of features estimated from the plurality of input images using a trained model; Equipped with The trained model is It includes an input layer that performs convolution, The method has been trained using training data including a plurality of training input images and a training feature-extracted image in which the features are extracted from one of the plurality of training input images; the positions of the features in the plurality of learning input images are different from one another; an amount of change in the position of the feature in the plurality of learning input images is smaller than a kernel size of a filter in the input layer.

14. The image processing method according to claim 13 , wherein a variation in position of the feature in the plurality of input images is less than a kernel size of a filter in the input layer.

15. an image processing unit that outputs an extracted image of features estimated from a plurality of input images using a trained model; The trained model is It includes an input layer that performs convolution, The method has been trained using training data including a plurality of training input images and a training feature-extracted image in which the features are extracted from one of the plurality of training input images; the positions of the features in the plurality of learning input images are different from one another; an amount of change in the position of the feature in the plurality of learning input images is smaller than a kernel size of a filter in the input layer.

16. a weld portion for welding together the workpieces; One or more image capture devices for capturing images of the welded portions of the workpieces; an image processing unit that outputs an extracted image of welding features estimated from a plurality of images captured by the imaging device using a learning model; a control unit that controls the welding portion based on the feature extraction image output by the image processing unit; Equipped with The trained model is It includes an input layer that performs convolution, The method has been trained using training data including a plurality of training input images and a training feature-extracted image in which the features are extracted from one of the plurality of training input images; the positions of the features in the plurality of learning input images are different from one another; A welding system, wherein a change in the position of the feature in the plurality of learning input images is smaller than a kernel size of a filter of the input layer.

Citation Information

Patent Citations

  • Vertical welding arc point recognizing device and method thereof

    JP1996132230A

  • Automatic welding system, welding control method, and machine learning model

    JP2018192524A

  • Laser processing system

    JP2019141902A

  • Machine learning program, machine learning method, and machine learning device

    JP2019185483A

  • Weld control device, display controller, welding system, weld control method and program

    JP2020028889A