Determination device, determination system, welding system, determination method and storage medium

Abstract

According to one embodiment, the determination device includes a processing unit. When a first image obtained by photographing a welding part during welding is input, the processing unit inputs the first image into a first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the processing unit uses the first image to determine whether welding is appropriate or not, and when the first model determines that the first image is inappropriate, the processing unit outputs information for correcting the photographing conditions of the welding part.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to a determination device, a determination system, a welding system, a determination method, and a storage medium Background Art

[0002] There is a determination device for determining the quality of welding, and the determination accuracy is desired to be high. Summary of the invention

[0003] According to one embodiment, the determination device includes a processing unit. When a first image obtained by photographing a welding part during welding is input, the processing unit inputs the first image into a first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the processing unit uses the first image to determine whether welding is appropriate or not, and when the first model determines that the first image is inappropriate, the processing unit outputs information for correcting the photographing conditions of the welding part. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 This is a schematic diagram illustrating the configuration of a welding system including the determination device according to the embodiment.

[0005] Figure 2 It is a schematic diagram illustrating the functional configuration of the determination device according to the embodiment.

[0006] Figure 3 This is a flowchart illustrating the operation of the welding system according to the embodiment.

[0007] Figure 4 This is a schematic diagram illustrating the welding process in the first embodiment.

[0008] Figure 5 This is an example of an image obtained in the first embodiment.

[0009] Figure 6 This is an example of an image obtained in the first embodiment.

[0010] Figure 7 This is a schematic diagram illustrating the welding process in the second embodiment.

[0011] Figure 8 This is an example of an image obtained in the second embodiment.

[0012] Fig. 9 This is a schematic diagram illustrating the welding process in the third embodiment.

[0013] Fig.10 This is an example of an image obtained in the third embodiment.

[0014] Fig.11This is an example of an image obtained in the third embodiment.

[0015] Fig.12 It is a schematic diagram illustrating the functional configuration of a determination device according to a first modified example of the embodiment.

[0016] Fig.13 It is a schematic diagram illustrating the functional configuration of a determination device according to a second modified example of the embodiment.

[0017] Fig.14 This is an image illustrating the processing of the determination device according to the second modified example of the embodiment.

[0018] Fig.15 This is an image illustrating the processing of the determination device according to the second modified example of the embodiment.

[0019] Fig.16 This is an image illustrating the processing of the determination device according to the second modified example of the embodiment.

[0020] Fig.17 It is a schematic diagram illustrating the functional configuration of a determination device according to a third modified example of the embodiment.

[0021] Fig.18 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0022] Fig.19 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0023] Fig. 20 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0024] Fig.21 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0025] Fig. 22 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0026] Fig.23 This is an image illustrating the processing of the determination device according to the third modified example of the embodiment.

[0027] Fig.24 This is a schematic diagram illustrating another welding system including the determination device according to the embodiment.

[0028] Fig.25 This is a flowchart illustrating the operation of the learning device.

[0029] Fig.26 This is a schematic diagram illustrating the hardware configuration of the determination device according to the embodiment. DETAILED DESCRIPTION

[0030] According to one embodiment, the determination device includes a processing unit. When a first image obtained by photographing a welding part during welding is input, the processing unit inputs the first image into a first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the processing unit uses the first image to determine whether welding is appropriate or not, and when the first model determines that the first image is inappropriate, the processing unit outputs information for correcting the photographing conditions of the welding part.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the size between parts, etc. are not necessarily the same as in reality. Even when the same part is shown, the sizes and ratios may be shown differently depending on the drawings.

[0033] In the present specification and each drawing, the same reference numerals are given to the same elements as those already described, and detailed description thereof will be appropriately omitted.

[0034] Figure 1 This is a schematic diagram illustrating the configuration of a welding system including the determination device according to the embodiment.

[0035] The welding system 1 includes a determination device 10 , a welding device 20 , and a storage device 30 .

[0036] The welding device 20 welds two or more parts to form an integral body. The welding device 20 performs, for example, arc welding or laser welding. Specifically, arc welding is Tungsten Inert Gas (TIG) welding, Metal Inert Gas (MIG) welding, Metal Active Gas (MAG) welding, or carbon dioxide gas arc welding. Here, an example in which the welding device 20 performs TIG welding will be mainly described.

[0037] Welding device 20 includes, for example, a head 21 , an arm 22 , a power supply unit 23 , a gas supply unit 24 , a welding wire 25 , an imaging unit 26 , an illumination unit 27 , and a control unit 28 .

[0038] An electrode 21a made of tungsten is provided on the head 21. The tip of the electrode 21a is exposed from the head 21. For example, the head 21 is attached to a multi-jointed arm 22 including a plurality of links. Alternatively, the head 21 may be provided on a welding torch held by an operator.

[0039] The power supply unit 23 is electrically connected to the electrode 21a and the welding object S. The power supply unit 23 applies a voltage between the electrode 21a and the welding object S to generate arc discharge. Alternatively, one of the electrode 21a and the welding object S may be set to a common potential (for example, a ground potential), and the power supply unit 23 may control only the potential of the other of the electrode 21a and the welding object S.

[0040] The gas supply unit 24 is connected to the head 21. The gas supply unit 24 supplies an inert gas to the head 21. Alternatively, the gas supply unit 24 may supply a mixed gas of an inert gas and an active gas. The gas supplied to the head 21 is ejected toward the welding object S from the front end of the head 21 where the electrode 21a is exposed.

[0041] The tip of the welding wire 25 is arranged in a space where arc discharge is generated. The tip of the welding wire 25 melts due to the arc discharge and drips toward the welding object S. The molten welding wire 25 solidifies, and the welding object S is welded. The welding wire 25 is fixed to the arm 22, for example, and is automatically supplied in accordance with the progress of melting.

[0042] The camera unit 26 captures the welded area during welding. The camera unit 26 captures the welded area to obtain a static image. Alternatively, the camera unit 26 may capture a dynamic image. The camera unit 26 cuts out a portion of the dynamic image to obtain a static image. The camera unit 26 is, for example, a camera including a CCD image sensor or a CMOS image sensor.

[0043] The illumination unit 27 illuminates the welded portion during welding so that a clearer image can be obtained by the imaging unit 26. Even if the welded portion is not illuminated, the illumination unit 27 may not be provided as long as an image that can be used for determination described later can be obtained.

[0044] The control unit 28 controls the operation of the above-mentioned components of the welding device 20. For example, the control unit 28 generates arc discharge while driving the arm unit 22, and welds the welding object S in a predetermined direction. In addition, the control unit 28 may control the settings of the imaging unit 26, the lighting unit 27, and the like.

[0045] The control unit 28 stores the image acquired by the imaging unit 26 in the storage device 30. For example, the control unit 28 stores the captured image, the welding conditions at the time of the image capture, and the image capture conditions at the time of the image capture in the storage device 30 in association with each other.

[0046] Welding conditions include, for example, applied voltage, gas flow rate, current value, wire feeding speed, or welding speed. Shooting conditions include, for example, exposure time, aperture, or sensitivity (ISO) settings of the camera unit 26. Shooting conditions may also include settings of the lighting unit 27. For example, when a pulse current is supplied to the lighting unit 27, the shooting conditions also include pulse width, pulse frequency, duty cycle, or peak value. In addition, here, when multiple items are listed by connecting "or", all of these items may be included, or only part of these items may be included.

[0047] Figure 2 It is a schematic diagram illustrating the functional configuration of the determination device according to the embodiment.

[0048] The determination device 10 includes a processing unit 11. The processing unit 11 includes, for example, an input unit 11a, a first determination unit 11b, an extraction unit 11c, a second determination unit 11d, and an output unit 11e.

[0049] The determination device 10 is connected to the welding device 20 and the storage device 30 by wire or wirelessly. The determination device 10 may be connected to the welding device 20 and the storage device 30 via a network. The determination device 10, the welding device 20, and the storage device 30 may be assembled into one device.

[0050] The input unit 11a accesses the storage device 30 to obtain the image IM stored in the storage device 30. Alternatively, the image may be directly sent from the control unit 28 to the processing unit 11. The input unit 11a receives data sent from the control unit 28.

[0051] When an image during welding is input to the input unit 11a, the first determination unit 11b refers to the learned first model M1. The first model M1 is used to determine whether the image captured during welding is appropriate. The first determination unit 11b inputs the image to the learned first model M1 and determines whether the image is appropriate. When the first model M1 determines that the image is inappropriate, the first model M1 also outputs information for correcting the imaging conditions.

[0052] For example, the first model M1 is a neural network. The first model M1 is pre-learned using learning data. The learning data includes a plurality of images captured during welding and a plurality of labels. The plurality of labels respectively indicate whether the plurality of images are appropriate. In addition, for inappropriate images, the plurality of labels indicate how the images are inappropriate.

[0053] For example, the first model M1 determines whether the image is too dark or too bright. If the first model M1 determines that the image is too dark or too bright, the first determination unit 11b outputs information indicating this. The first model M1 may also output a specific correction amount for shooting conditions for shooting a more appropriate image.

[0054] As an example, the first model M1 determines that the image is too bright, and outputs information indicating this. The output unit 11e sends the information to the control unit 28. Upon receiving the information, the control unit 28 changes the shooting conditions so that the image becomes darker. For example, the control unit 28 shortens the exposure time, increases the aperture, reduces the sensitivity, shortens the pulse width, reduces the pulse frequency, reduces the duty cycle, or reduces the peak value. The control unit 28 re-shoots the welded portion during welding under the changed shooting conditions.

[0055] When the first model M1 determines that the image is appropriate, the extraction unit 11c refers to the learned second model M2. The second model M2 is used to extract welding-related features from the image. For example, the second model M2 is a neural network. The second model M2 is pre-learned using learning data. The learning data includes a plurality of appropriate images and a plurality of images each representing features based on the images.

[0056] The extraction unit 11c inputs an image to the learned second model M2 to extract features related to welding. When extracting features, the second determination unit 11d refers to the learned third model M3. The third model M3 is used to determine whether welding is appropriate. The third model M3 is stored in the storage device 30 or other storage devices. The second determination unit 11d inputs an image representing features to the third model M3 to determine whether welding represented by the image is appropriate.

[0057] For example, the third model M3 is a neural network. The third model M3 is learned in advance using learning data. The learning data includes a plurality of images showing characteristics related to welding and a plurality of labels indicating whether the welding shown in these images is appropriate or not.

[0058] The third model M3 may further output a correction amount of welding conditions when it is determined that welding is inappropriate. In this case, each label of the learning data indicates whether welding is appropriate or not, and indicates a correction amount for inappropriate welding. By learning using labels indicating correction amounts, the correction amount of welding conditions can be output from the third model M3.

[0059] Alternatively, the third model M3 may be learned to output only the correction amount instead of the appropriateness of welding. In this case, when the correction amount output by the third model M3 is zero or very small, it can be considered that the welding is determined to be appropriate by the third model M3.

[0060] The output unit 11e outputs the determination result of the second determination unit 11d to the outside. For example, when the correction amount is output by the third model M3, the output unit 11e sends the correction amount to the control unit 28. The control unit 28 changes the welding conditions according to the correction amount.

[0061] As an example, when the welding position deviates from the standard position, the third model M3 determines that the welding is inappropriate. The welding position corresponds to the position of the electrode 21a. In this case, the third model M3 outputs the correction amount of the position of the electrode 21a for more appropriate welding.

[0062] The output unit 11e may associate the determination result with the welding conditions when the image used for determination is obtained. The output unit 11e stores the associated information IN in the storage device 30. For example, the welding device 20 refers to the information IN in the storage device 30 during welding. The welding device 20 adjusts the welding conditions based on the information IN so that more appropriate welding can be performed.

[0063] For example, a plurality of determination results and a plurality of welding conditions are stored in association with each other in the storage device 30. Such a collection of associated information is called a process map. By accumulating determination results and welding conditions, the quality of welding performed by the welding device 20 can be improved.

[0064] The effects of the embodiment will be described.

[0065] In arc welding and laser welding, strong light is generated locally during welding. Therefore, if the shooting conditions are not set properly, it is impossible to obtain an image that can be used to determine whether the welding is appropriate. In addition, the image obtained also changes according to the welding conditions such as the groove width, the shape of the groove, the welding position in the groove, and the shape of the parts during fillet welding. Therefore, the captured image is prone to deviation. For example, if the captured image is too bright, the appearance of the welding cannot be fully expressed in the image, and it is impossible to determine whether the welding is appropriate based on the image.

[0066] In view of this problem, in the embodiment, the processing unit 11 of the determination device 10 determines whether the image captured by the imaging unit 26 is appropriate using the first model. As a result, it is possible to determine whether welding is appropriate or not using only the image determined to be appropriate by the first model. For example, an image that is too bright or too dark is not used to determine whether welding is appropriate or not. As a result, it is possible to determine whether welding is appropriate or not with higher accuracy.

[0067] Furthermore, when the first model determines that the image is inappropriate, the processing unit 11 outputs information for correcting the imaging conditions. Based on this information, the imaging conditions of the imaging unit 26 are corrected. This makes it easy to obtain an image useful for determining whether welding is appropriate or not.

[0068] In addition, when the processing unit 11 determines that the image is appropriate by the first model, it extracts features related to welding from the image using the second model. The processing unit 11 uses the extracted features to determine whether the welding is appropriate or not. Specifically, the processing unit 11 inputs the extracted features into the third model to determine whether the welding represented by the features is appropriate or not. According to this method, the accuracy of the determination can be improved compared to the case where the appropriateness of the welding is determined based on the entire image.

[0069] As examples of other determination methods, there are the following methods.

[0070] The storage device 30 stores a comparison image of a good welding state in advance. When the processing unit 11 determines that the image is appropriate by the first model, it compares the image with the comparison image. For example, when the similarity between these images is high, the processing unit 11 determines that the welding is good. The similarity can be represented by the difference in feature quantity. The processing unit 11 determines whether the welding is appropriate or not by comparing the difference in feature quantity with a preset threshold value.

[0071] This method can also determine whether welding is appropriate. However, it is more preferable that the processing unit 11 uses the second model and the third model to determine whether welding is appropriate. By extracting features related to welding from the image using the second model, it is possible to determine whether welding is appropriate with higher accuracy.

[0072] Figure 3 This is a flowchart illustrating the operation of the welding system according to the embodiment.

[0073] First, the camera unit 26 captures the welded part during welding and obtains an image (first image) (step S1). The first determination unit 11b of the processing unit 11 uses the first model to determine whether the image is appropriate (step S2). If the image is not appropriate, information is sent from the output unit 11e to the control unit 28 to correct the shooting conditions (step S3).

[0074] If the image is appropriate, the extraction unit 11c extracts the feature (first feature) related to welding from the image using the second model (step S4). The second determination unit 11d determines whether the welding indicated by the feature is appropriate or not using the third model (step S5). If the welding is not appropriate, information is sent from the output unit 11e to the control unit 28 to correct the welding conditions (step S6).

[0075] When welding is determined to be appropriate or welding conditions are corrected, the output unit 11e outputs information (step S7). For example, the output unit 11e associates the determination result with the captured image, the welding conditions at the time of capturing, and the capturing conditions at the time of capturing, and stores them in the storage device 30.

[0076] (First embodiment)

[0077] Figure 4 This is a schematic diagram illustrating the welding process in the first embodiment.

[0078] Figure 5 (a) to Figure 5 (i) Figure 6 (a) and Figure 6 (b) is an example of an image obtained in the first embodiment.

[0079] In the first embodiment, Figure 4 The first member 91a and the second member 92a shown are TIG welded. The first member 91a has a first wall surface SW1 provided at the groove. The second member 92a has a second wall surface SW2 provided at the groove.

[0080] The tip of the electrode 21a and the tip of the conductor wire 25 are arranged between the first wall surface SW1 and the second wall surface SW2, and arc discharge is generated. The welding wire 25 melts to form a molten pool WP. The molten pool WP solidifies, and the first member 91a and the second member 92a are welded.

[0081] The imaging unit 26 captures the molten pool WP, the tip of the electrode 21a, and the tip of the lead wire 25 from an angle where the molten pool WP, the tip of the electrode 21a, and the tip of the lead wire 25 can be seen. Figure 4 When the image is acquired by the imaging unit 26, the first determination unit 11b of the processing unit 11 determines whether the image is appropriate using the first model.

[0082] Figure 5 (a) to Figure 5 (c) is an example of an image determined to be too dark. Figure 5 (d) to Figure 5 (f) is an example of an image determined to be appropriate. Figure 5 (g) to Figure 5 (i) is an example of an image determined to be too bright.

[0083] If you will Figure 5 (a) to Figure 5 If the image shown in (c) is input to the first model, the first model will output the correction amount of the shooting condition used to make the image brighter together with the judgment result indicating that the image is inappropriate. Figure 5 (g) to Figure 5 When the image shown in (i) is input to the first model, the first model outputs the correction amount of the shooting condition for making the image darker together with the determination result indicating that the image is inappropriate.

[0084] Figure 6 (a) represents an image determined to be appropriate by the first model. The extraction unit 11c Figure 6 The image shown in (a) is input to the second model. As a result, Figure 6 The line segments indicated by the dashed line BL and the dotted line DL in the image (b) are extracted as features. Figure 6 In the example (b), the extracted line segments are compared with Figure 6 The image of (a) is displayed in an overlapping manner. Not limited to this example, the second model M2 may output an image that does not include the original image but includes only the extracted line segments.

[0085] exist Figure 6 In the image shown in (b), the dashed lines BL correspond to the first wall surface SW1 and the second wall surface SW2, respectively. The dotted line DL corresponds to the end of the molten pool WP. The second determination unit 11d Figure 6 The image shown in (b) is input to the third model to determine whether welding is appropriate.

[0086] exist Figure 6 In the image shown in (b), both ends of the dotted line DL are connected to the dotted line BL. This means that welding is performed in the center between the first wall surface SW1 and the second wall surface SW2. For example, Figure 6 The welding shown in the image of (b) was judged to be appropriate.

[0087] On the other hand, if the end of the dotted line DL is separated from the dotted line BL, or the center of the dotted line DL is offset from the center of the dotted lines BL in the direction connecting the dotted lines BL, the welding is determined to be inappropriate. For example, the movement amount of the third model output electrode 21a is such that welding is performed at an appropriate position.

[0088] (Second embodiment)

[0089] Figure 7 This is a schematic diagram illustrating the welding process in the second embodiment.

[0090] Figure 8 (a) and Figure 8 (b) is an example of an image obtained in the second embodiment.

[0091] In the second embodiment, Figure 7 The first member 91b and the second member 92b shown are MIG welded. The first member 91b and the second member 92a have a first wall surface SW1 and a second wall surface SW2 provided in the groove, respectively.

[0092] The tip of the welding wire 25 is drawn out from the electrode 21a and arranged between the first wall surface SW1 and the second wall surface SW2. Arc discharge is generated between the welding wire 25 and the welding object, melting the welding wire 25. Thus, a molten pool WP is formed, and the first member 91b and the second member 92b are welded.

[0093] The camera unit 26 captures the image from an angle where the molten pool WP and the tip of the welding wire 25 can be seen. Figure 7 The welding is shown. Figure 8 (a) represents an image determined to be appropriate by the first model. The extraction unit 11c Figure 8 The image shown in (a) is input to the second model. As a result, Figure 8 The line segments indicated by the dotted line BL, the dotted line DL, and the dashed-dotted line CL in the image of (b) are extracted as features.

[0094] exist Figure 8 In the image shown in (b), the dotted lines BL correspond to the first wall surface SW1 and the second wall surface SW2. The dotted line DL corresponds to the end of the molten pool WP. The dot-dash line CL corresponds to the front end of the welding wire 25. The second determination unit 11d Figure 8 The image shown in (b) is input to the third model to determine whether welding is appropriate.

[0095] exist Figure 8 In the image shown in (b), the dotted line DL is located in the center between the dotted lines BL in the direction connecting the dotted lines BL. Figure 8 The welding shown in the image of (b) was judged to be appropriate.

[0096] (Third embodiment)

[0097] Fig. 9 This is a schematic diagram illustrating the welding process in the third embodiment.

[0098] Fig.10 (a) to Fig.10 (d) Fig.11 (a) and Fig.11 (b) is an example of an image obtained in the third embodiment.

[0099] In the third embodiment, Fig. 9 The first member 91c and the second member 92c shown are laser welded. The first member 91c and the second member 92c are placed in contact with each other, and the laser light L is irradiated on the contact portion.

[0100] When irradiated with the laser light L, the first component 91c and the second component 92c melt to form a molten pool WP. When the molten pool WP is cooled, a solidified layer SL is formed. The first component 91c and the second component 92c are welded by sequentially irradiating the contacting portions of the first component 91c and the second component 92c with the laser light L.

[0101] The imaging unit 26 captures the molten pool WP and the irradiation point of the laser light L from an angle where the molten pool WP and the irradiation point of the laser light L can be seen. Fig. 9 For example, the camera unit 26 obtains Fig.10 The first determination unit 11b Fig.10The image shown in (a) is input to the first model to determine whether it is appropriate.

[0102] For example, Fig.10 The image shown in (a) is determined to be inappropriate because it is too dark. The output unit 11e sends the correction amount of the shooting condition to the control unit 28 so that the image becomes brighter.

[0103] Under the changed shooting conditions, the camera unit 26 takes another shot of the welding state. Fig.10 The first determination unit 11b Fig.10 The image shown in (b) is input to the first model M1 and its suitability is judged again.

[0104] The first model M1 may be learned to determine whether an image is appropriate based on a portion of the image. Fig.10 (c) and Fig.10 (d) is an example of an image obtained by the imaging unit 26. Fig.10 (c) and Fig.10 In (d), the dark part on the right is the molten pool (liquid), and the other bright parts are the part and the solidified layer (solid).

[0105] For example, in the judgment based on the first model, the contrast of the image has an influence. The image is judged to be too bright or too dark based on the difference between the maximum brightness and the minimum brightness in the image. It is preferable to clearly reflect the molten pool in the image. Therefore, it is preferable to judge whether the image is appropriate based on the minimum brightness of the part based on the molten pool and the maximum brightness of the other solid parts.

[0106] For example, Fig.10 As shown by the dotted line BL in (c), it is preferred to determine whether the image is appropriate based on only a portion of the image including a portion of the molten pool and a portion of the solid portion. This can suppress the reduction in the accuracy of image determination due to unintentional black spots or white spots included in the image.

[0107] In addition, if Fig.10 (c) and Fig.10 As shown in (d), the size and position of the molten pool can vary for each image. Therefore, the first model M1 is preferably learned to be able to extract the part used for judgment for each image. Fig.10 In the image shown in (d), the position and size of the dotted line BL are Fig.10 The position and size of the dotted line BL in (c) are different.

[0108] Fig.11 (a) represents an image determined to be appropriate by the model M1 after the above processing. The extraction unit 11c Fig.11The image shown in (a) is input to the second model. As a result, Fig.11 The line segments indicated by the dotted line BL, the dotted line DL, and the dashed-dotted line CL in the image of (b) are extracted as features.

[0109] exist Fig.11 In the image shown in (b), the dotted line BL corresponds to the outer edge of the solidified layer SL. The dotted line DL corresponds to the outer edge of the molten pool WP. The dashed-dotted line CL corresponds to the outer edge of the irradiation point of the laser L. The second determination unit 11d Fig.11 The image shown in (b) is input to the third model to determine whether welding is appropriate.

[0110] exist Fig.11 In the image shown in (b), the shape of the dot-dash line CL is close to a perfect circle, so the welding is judged to be appropriate. In addition, the appropriateness of welding can also be judged based on the change of the size or shape of the dot-dash line CL at each time, the change of the length of the dotted line DL in a predetermined direction at each time, the change of the distance between the dotted lines BL at each time, the change of the distance between the dotted lines BL in one image, etc.

[0111] When judging whether welding is appropriate or not based on the change with the passage of time, the third model M3 is also learned accordingly. In this case, for example, the third model M3 is a recurrent neural network (RNN). Preferably, the third model M3 has a Long Short-Term Memory (LSTM) structure. For example, the third model M3 is learned using time series data and a label indicating whether the time series data is appropriate or not.

[0112] (First Modification)

[0113] Fig.12 It is a schematic diagram illustrating the functional configuration of a determination device according to a first modified example of the embodiment.

[0114] In the determination device 10a of the first modified example, the processing unit 11 does not include the extraction unit 11c. When the image is determined to be appropriate by the first model M1, the second determination unit 11d directly determines the appropriateness of welding based on the image using the third model M3 stored in the storage device 30. In addition, when the welding is determined to be inappropriate by the third model M3, the correction amount of the welding condition is output.

[0115] Thus, in the determination device 10a of the first modified example, features are not extracted, and the suitability of welding is directly determined by the third model M3 based on the image determined to be suitable by the first model M1. Figure 2 Compared with the determination device 10 shown in FIG. 1 , the amount of calculation can be reduced. Figure 2In the judgment device 10 shown, different models are used to extract features from images and to judge welding. Therefore, even if an abnormality occurs in the output from the processing unit 11, the models can be inspected, adjusted or replaced. That is, the maintenance of the used models becomes easy.

[0116] (Second Modification)

[0117] Fig.13 It is a schematic diagram illustrating the functional configuration of a determination device according to a second modified example of the embodiment.

[0118] In the determination device 10b of the second modified example, when the extraction unit 11ca extracts features using the second model M2, it directly extracts features for the corrected image or extracts features for the corrected image. For example, when the image input to the second model M2 is rotated, enlarged, or reduced relative to the reference state, the second model M2 outputs features for the image after the input image is rotated, enlarged, or reduced.

[0119] In addition, the second model M2 may also output features for an image after contrast adjustment of the input image. When the input image contains defects, damage, or weld bead marks of the base, the second model M2 extracts features for an image after the defects, damage, or weld bead marks are removed. Alternatively, the second model M2 may also output features for an image after the weld bead width of the input image is enlarged or reduced. That is, correction of the image may include rotation, enlargement, reduction, contrast adjustment, removal of defects, removal of damage, removal of weld bead marks, or change of weld bead width.

[0120] For example, if an image that does not contain defects, damage, or weld bead marks is used when learning the second model M2, then when an image containing defects, damage, or weld bead marks is input, the second model M2 may not be able to properly extract features related to welding. In addition, if the image input to the second model is rotated, enlarged, or reduced compared to the image used for learning, it may not be possible to properly extract features related to welding. Or, even if features related to welding can be extracted, it may not be possible to correctly judge whether welding is appropriate based on the extracted features by the third model M3 afterwards.

[0121] As described above, the second model M2 can output features for an image obtained by correcting an input image, and thus can determine the suitability of welding with higher accuracy based on the extracted features.

[0122] Fig.14 (a) to Fig.14 (d) Fig.15 (a) to Fig.15 (d) Fig.16 (a) and Fig.16 (b) is an image illustrating the processing of the determination device according to the second variant of the embodiment.

[0123] exist Fig.14 (a) to Fig.14 (d) Fig.15 (a) to Fig.15 (d) Fig.16 (a) and Fig.16 (b) shows the state of TIG welding.

[0124] Fig.14 In the image (a), the size of the melt pool in the example image is larger than the standard one. Fig.14 The image shown in (b) is rotated relative to the standard image. Fig.14 The image shown in (c) is judged to be appropriate by the first model M1, but the contrast is higher than the standard image. Fig.14 (a) to Fig.14 The images shown in (c) are input to the second model, and the output is a rotation, reduction or contrast adjustment of the feature output directly extracted from these images. Fig.14 The features shown in (d).

[0125] exist Fig.15 In the image shown in (a), a defect DF is shown. The second model outputs a Fig.15 The characteristics of the image shown in (b). Fig.15 In the image shown in (c), the bead width W is wider than the standard. The second model outputs the reduced bead width W. Fig.15 Features of the image shown in (d).

[0126] As another example, in Fig.16 In the image shown in (a), there is damage F. Fig.16 In the image shown in (b) of FIG. 1 , a weld bead mark B is shown. The second model outputs features based on the image from which the damage F and the weld bead mark B have been removed.

[0127] (Third Modification)

[0128] Fig.17 It is a schematic diagram illustrating the functional configuration of a determination device according to a third modified example of the embodiment.

[0129] In the determination device 10c of the second modified example, the processing unit 11 includes an evaluation unit 11f. The evaluation unit 11f evaluates the features output by the extraction unit 11c.

[0130] Specifically, if the feature is extracted by the extraction unit 11c, the evaluation unit 11f corrects the feature. Correction includes supplementing the feature, deleting unnecessary parts, or clarifying the feature. The evaluation unit 11f compares the extracted feature with the corrected feature. The smaller the difference between these features, the higher the performance of the second model M2 is evaluated by the evaluation unit 11f.

[0131] As an example, the evaluation unit 11f calculates the difference between an image representing the extracted feature (first feature) and an image representing the corrected feature (second feature). The difference is obtained by comparing the corresponding pixels of the two images with each other and counting the number of changed pixels. The evaluation unit 11f compares the number of changes with a preset threshold value (first threshold value). When the number of changes is greater than the threshold value, the evaluation unit 11f determines that it is desired to relearn or replace the second model M2. For example, information requesting relearning or replacement of the second model M2 is output to the user from the output unit 11e.

[0132] The second model M2 may fail to appropriately extract features from the image when the tendency of the image acquired by the imaging unit 26 changes, etc. By evaluating the second model M2 together with the determination of the suitability of welding based on the image, a decrease in determination accuracy can be suppressed.

[0133] Fig.18 (a) Fig.18 (b) Fig.19 (a) to Fig.19 (f), Fig. 20 (a) to Fig.21 (c) Fig. 22 (a) to Fig. 22 (c) and Fig.23 (a) to Fig.23 (c) is an image illustrating the processing of the determination device according to the third variant of the embodiment.

[0134] exist Fig.18 (a) to Fig.23 (c) shows the state of TIG welding.

[0135] Fig.18 (a) is an example of an image determined to be appropriate by the first model. The extraction unit 11c Fig.18 The image shown in (a) is input to the second model. Thus, Fig.18 The characteristics shown in (b).

[0136] exist Fig.18 The image shown in (b) includes two lines L1 and L2 corresponding to the wall surface of the groove, and a line L3 corresponding to the end of the molten pool. Fig.18In the image (b), lines L1 and L2 are indicated in red. Line L3 is indicated in green. The other parts are indicated in blue. Fig.18 The image shown in (b) is corrected. For example, the evaluation unit 11f deletes or supplements the lines representing the features during the correction, and removes the noise included in the image. A specific example of the correction is described below.

[0137] Fig.19 (a) Fig.19 (b) and Fig.19 (c) The image is divided by color Fig.18 The image shown in (b). Fig.19 The image of (a) contains only lines L1 and L2. Fig.19 The image of (b) contains only line L3. Fig.19 The image (c) contains only other elements. Fig.19 The image in (c) contains essentially no content.

[0138] The evaluation unit 11f performs binarization on each image. Binarization is performed so that the brightness of lines L1 to L3 is different from the brightness of other parts. The evaluation unit 11f applies at least one of reduction processing and enlargement processing to the binary image. For example, the evaluation unit 11f applies both reduction processing and enlargement processing to the binary image. The evaluation unit 11f may perform enlargement processing and reduction processing multiple times. The enlargement processing and reduction processing may also be repeated alternately. The execution order of the enlargement processing and the reduction processing may be changed appropriately. The evaluation unit 11f may add a rotation processing when performing the reduction processing or the enlargement processing. Thus, the generated Fig.19 (d) to Fig.19 The image shown in (f) is shown in FIG. By reducing and enlarging the image, the noise in the image is removed.

[0139] like Fig. 20 As shown in (a), the evaluation unit 11f sets two regions (region R1 and region R2) in the image including the lines L1 and L2. The boundary between the region R1 and the region R2 is the center of the first direction D1 of the image. The first direction D1 is the direction connecting the lines L1 and L2. In addition, the evaluation unit 11f calculates the total of the lightness and darkness values ​​in the second direction D2 at each point in the first direction D1 for the image including the lines L1 and L2. The second direction D2 is a direction perpendicular to the first direction D1.

[0140] Fig. 20 (b) shows the calculation result of the total of the light and dark values. Fig. 20In (b), the horizontal axis represents the position P in the first direction D1. The vertical axis represents the total Sum of the shading values. The evaluation unit 11f detects the peak of the total Sum of each of the regions R1 and R2. When multiple peaks are detected in the region R1 or R2, only the peak closer to the center of the first direction D1 of the image is retained, and the other peaks are removed. Thus, an image with only one line in each of the regions R1 and R2 is obtained.

[0141] For example, in the image output from the model, in addition to lines L1 and L2, there may be other lines along one direction. Through the above processing, the lines far from the center are removed. Fig. 20 (a) and Fig. 20 In the example (b), only one peak is detected in each of regions R1 and R2. Fig. 20 (a) and Fig. 20 In the example (b), no peak is deleted. Next, the evaluation unit 11f performs straight line approximation on the lines L1 and L2. Fig. 20 The image shown in (a) is obtained Fig. 20 The image shown in (c).

[0142] The evaluation unit 11f evaluates the position of the lines L1 and L2. Fig.19 The image shown in (e) is as follows: Fig.21 As shown in (a), five regions R11 to R15 are set. The position of the boundary between regions R11 and R12 corresponds to the position of line L1. The position of the boundary between regions R14 and R15 corresponds to the position of line L2. Regions R12 to R14 are set by dividing the region between the boundary corresponding to line L1 and the boundary corresponding to line L2 into three equal parts.

[0143] When part of line L3 exists in regions R11 and R15, the evaluation unit 11f removes the part. That is, the evaluation unit 11f sets the brightness of all pixels included in regions R11 and R15 to values ​​corresponding to black. In addition, the evaluation unit 11f performs curve approximation on part of line L3 that exists in regions R12 and R14. The evaluation unit 11f does not perform any processing on part of line L3 that exists in region R13. Through these processing, the generated Fig.21 The image shown in (b).

[0144] The evaluation department 11f will Fig.19 The image shown in (f) Fig. 20 The image shown in (c) and Fig.21 Combine the images shown in (b) to generate Fig.21 The image shown in (c). Fig.18In the image shown in (b), lines L1 to L3 are unclear, but these lines become clear after correction. Fig.18 The image shown in (b) is Fig.21 The evaluation unit 11f compares the image shown in (c) with the image shown in (c) and counts the number of pixel changes. The evaluation unit 11f evaluates the performance of the second model based on the number of changes.

[0145] Fig. 22 (a) to Fig. 22 (c) and Fig.23 (a) to Fig.23 (c) shows other examples.

[0146] The evaluation department 11f will Fig. 22 The image of (a) is input to the second model to obtain Fig. 22 (b) is an image of. Fig. 22 In the image (b), line L1 is unclear. Fig. 22 In the image (b), in addition to the lines L1 to L3, the line L4 is also included. Through correction, the line L1 becomes clear and the line L4 is removed. The evaluation unit 11f Fig. 22 The image of (b) is corrected by applying the same processing as above, generating Fig. 22 Image of (c).

[0147] The evaluation department 11f will Fig. 22 The image shown in (b) is Fig. 22 The evaluation unit 11f compares the image shown in (c) with the image shown in (c) and counts the number of pixel changes. The evaluation unit 11f evaluates the performance of the second model based on the number of changes.

[0148] Similarly, the evaluation unit 11f will Fig.23 The image of (a) is input to the second model to obtain Fig.23 The evaluation unit 11f evaluates Fig.23 The image (b) is corrected to generate Fig.23 The evaluation unit 11f will Fig.23 The image shown in (b) is Fig.23 The evaluation unit 11f compares the image shown in (c) with the image shown in (c) and counts the number of pixel changes. The evaluation unit 11f evaluates the performance of the second model based on the number of changes.

[0149] Fig.24 This is a schematic diagram illustrating another welding system including the determination device according to the embodiment.

[0150] Fig.24The welding system 2 shown includes a determination system including a determination device 10 and a learning device 40. The learning device 40 learns each model used by the determination device 10. The learning device 40 stores the learned model in the storage device 30.

[0151] As described above, supervised learning using the learning data is performed in each model. The learning device 40 may also perform semi-supervised learning. For example, when the second model M2 is trained, semi-supervised learning is used.

[0152] For example, the learning device 40 uses a plurality of input images and a plurality of teaching images representing features to cause the second model M2 to perform supervised learning (first action). Thereafter, the learning device 40 inputs other input images to the second model M2 to obtain an image output from the second model M2. The learning device 40 corrects the output image to obtain an appropriate image representing features related to welding. As described above, the correction includes supplementing features, deleting unnecessary parts, or clarifying features. The learning device 40 uses the other input images and the corrected images to cause the second model M2 to further perform supervised learning (second action).

[0153] In addition, the learning device 40 uses a plurality of input images and a plurality of teaching images to supervise the learning of the second model M2 (first action). The input image is, for example, rotated, enlarged, or reduced compared to the desired image. Alternatively, the input image includes defects, scars, etc. The teaching image is these corrected images. Thereafter, the learning device 40 inputs other input images to the second model M2 to obtain an image output from the second model M2. The learning device 40 corrects the output image to a more preferred state. The learning device 40 uses the other input images and the corrected images to further supervise the learning of the second model M2 (second action).

[0154] When learning the model, a teaching image is generated based on an image output from the model, thereby reducing the amount of data required for learning.

[0155] The learning device 40 may also have a function of generating an input image used in learning the second model M2. For example, the learning device 40 performs processing such as rotating, enlarging, reducing, changing the contrast, adding defects, adding damage, adding weld bead marks, changing the weld bead width, etc. on the teaching image used for learning to generate the input image. The learning device 40 uses the generated input image and the teaching image to learn the second model M2.

[0156] When learning a model, by generating an input image from a teaching image, the amount of data required for learning can be reduced.

[0157] Fig.25 This is a flowchart illustrating the operation of the learning device.

[0158] exist Fig.25 In the example, the semi-supervised learning operation performed by the learning device is illustrated. Fig.25 , an example is shown in which the learning device 40 receives evaluations of images used for learning from the user during semi-supervised learning.

[0159] The learning device 40 accesses the storage device 30 and inputs the first image (input image) to the learned model (step S11). The learning device 40 obtains the second image (output image) output from the model (step S2). The learning device 40 corrects the second image and generates a third image (corrected image) different from the second image (step S3).

[0160] The learning device 40 causes the display device to display the third image (step S14). The user confirms the third image and uses the input device to input an evaluation of the third image. The learning device 40 receives the evaluation from the user (step S15). The display device is, for example, a monitor or a projector. The input device is, for example, a keyboard, a mouse, a touch pad, or a microphone (sound input). A device having the functions of both a display device and an input device, such as a touch panel, may also be used.

[0161] The learning device 40 determines whether the evaluation is positive (step S16). When the evaluation is negative, the learning device 40 requests the user to input a teaching image. The learning device 40 accepts the teaching image (the fourth image) input from the user (step S17). The learning device 40 uses the first image as input data and the fourth image as teaching data to learn the model (step S18).

[0162] When the evaluation is positive, the learning device 40 calculates the difference between the second image and the third image. The learning device 40 determines whether the difference exceeds a predetermined threshold value (step S19). When the difference exceeds the threshold value, the learning device 40 uses the first image as input data and the third image as teaching data to learn the model (step S20). After step S18 or S20, the learning device 40 saves the learned model (step S21). When the difference is below the threshold value, the learning device 40 ends the processing.

[0163] according to Fig.25 In the operation shown, only the third image suitable as teaching data is used for model learning. In addition, when the third image is not suitable, the fourth image input by the user is used for model learning. Therefore, the model can be appropriately learned to output the result intended by the user.

[0164] exist Fig.25In the action shown, when the evaluation of the third image is positive, the learning device 40 may also execute step S20 regardless of the difference between the second image and the third image. However, when the difference is small, it means that a sufficiently ideal image is output from the model. Therefore, there is little need to perform learning using the third image. By judging the execution of learning based on the difference, it is possible to suppress unnecessary learning of the model. Thus, the amount of calculation can be reduced by the learning device 40. In addition, it is possible to suppress the model from being over-learned and the generalization ability of the model from being reduced.

[0165] Fig.26 This is a schematic diagram illustrating the hardware configuration of the determination device according to the embodiment.

[0166] Each of the determination devices includes a central processing unit (CPU) 111 , an input device 112 , an output device 113 , a ROM (Read Only Memory) 114 , a RAM (Random Access Memory) 115 , a storage device 116 , a communication device 117 , and a bus 118 . Each of the components is connected via the bus 118 .

[0167] The CPU 111 includes a processing circuit. The CPU 111 cooperates with various programs pre-stored in the ROM 114 or the storage device 116 to perform various processes and uniformly control the operation of the determination device 10. Thus, the function of the processing unit 11 in each of the above-mentioned determination devices is realized. During the processing, the CPU 111 uses a predetermined area of ​​the RAM 115 as a working area. The CPU 111 cooperates with the programs pre-stored in the ROM 114 or the storage device 116 to realize the input device 112, the output device 113, the communication device 117, etc.

[0168] The input device 112 includes, for example, a keyboard, a mouse, or a touch panel. The input device 112 receives information input from a user as an instruction signal and outputs the instruction signal to the CPU 111. The output device 113 is, for example, a monitor. The output device 113 outputs various information in a visually identifiable manner based on the signal output from the CPU 111.

[0169] ROM 114 stores programs and various setting information used in the control of the determination device 10 in a non-rewritable manner. RAM 115 is a volatile storage medium such as SDRAM (Synchronous Dynamic Random Access Memory). RAM 115 functions as a work area of ​​CPU 111. Specifically, it functions as a buffer for temporarily storing various variables and parameters used by the determination device 10.

[0170] The storage device 116 is a rewritable recording device such as a semiconductor storage medium such as a flash memory, a storage medium capable of recording magnetically or optically, etc. The storage device 116 stores programs and various setting information used in the control of the determination device 10. The communication device 117 is used to communicate with external devices to send and receive information.

[0171] Here, the hardware configuration of the determination device is described. The same configuration can also be applied to the hardware configuration of the learning device. Alternatively, Fig.26 The single piece of hardware shown may function as both the determination device and the learning device.

[0172] The embodiments may include the following configurations.

[0173] (Composition 1)

[0174] A determination device includes a processing unit, the processing unit performing the following processing:

[0175] When a first image obtained by photographing a welding portion during welding is input, the first image is input to a first model for determining whether the image is appropriate.

[0176] When the first model determines that the first image is appropriate, the appropriateness of welding is determined using the first image, and when the first model determines that the first image is inappropriate, information for correcting the imaging conditions of the welding part is output.

[0177] (Composition 2)

[0178] According to the determination device described in configuration 1, the processing unit, when determining that the first image is appropriate through the first model, inputs the first image into the second model for extracting features related to welding from the image, and uses the first feature extracted through the second model to determine whether the welding is appropriate.

[0179] (Composition 3)

[0180] The determination device according to Configuration 2, wherein the processing unit evaluates the performance of the second model using a difference between the first feature and a second feature obtained by correcting the first feature.

[0181] (Composition 4)

[0182] The determination device according to configuration 3, wherein the processing unit outputs information requesting relearning or replacement of the second model when the difference is larger than a preset first threshold value.

[0183] (Composition 5)

[0184] According to any one of configurations 2 to 4, the processing unit determines the suitability of welding represented in the first image by inputting the first feature into a third model for determining the suitability of welding based on features related to welding.

[0185] (Composition 6)

[0186] The determination device according to configuration 5, wherein when it is determined that welding is inappropriate, the third model further outputs a correction amount of welding conditions.

[0187] (Composition 7)

[0188] According to any one of configurations 2 to 6, the second model is capable of correcting the image and outputting features for the corrected image.

[0189] (Composition 8)

[0190] The determination device according to configuration 7, wherein the correction includes rotation, enlargement, reduction, or contrast adjustment of the image.

[0191] (Composition 9)

[0192] The determination device according to configuration 7 or 8, wherein the correction includes removal of defects, damage or weld bead marks.

[0193] (Composition 10)

[0194] The determination device according to any one of Configurations 7 to 9, wherein the correction includes enlarging or reducing the weld bead width.

[0195] (Composition 11)

[0196] A determination system comprising:

[0197] The determination device according to any one of 2 to 10; and

[0198] A learning device enables the second model to learn,

[0199] The learning device performs a first action and a second action,

[0200] The first operation is to use a plurality of input images and a plurality of teaching images to allow the second model to learn.

[0201] The second action is to cause the second model to learn using another input image and a corrected image obtained by inputting the other input image to the second model and correcting an output image output from the second model.

[0202] (Composition 12)

[0203] A welding system comprising:

[0204] The determination device according to any one of 1 to 10; and

[0205] The welding device includes an imaging unit for acquiring the first image.

[0206] According to the embodiments described above, it is possible to provide a determination device, a determination system, a welding system, and a determination method that can determine welding quality with higher accuracy. By performing welding based on the determination results after these are executed, it is possible to provide a welding device that can perform better welding. In addition, by using a program for causing a processing unit (processing circuit) to execute each embodiment described above and a storage medium storing the program, it is possible to perform welding determination with higher accuracy.

[0207] Several embodiments of the present invention are illustrated above, but these embodiments are presented as examples and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, changes, etc. can be made without departing from the scope of the main purpose of the invention. These embodiments and their modifications are included in the scope or main purpose of the invention, and are included in the invention described in the claims and the scope of their equivalents. In addition, the above-mentioned embodiments can be combined with each other and implemented.

Claims

1. A determination device comprising a processing unit, The processing unit performs the following processing: When a first image obtained by photographing a welding portion during welding is input, the first image is input to a first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the first image is input to a second model for extracting features related to welding from an image, and the first feature extracted by the second model is input to a third model for determining whether welding is appropriate, thereby determining whether welding shown in the first image is appropriate, and when the first model determines that the first image is inappropriate, outputting information for correcting the shooting conditions of the welding part, The processing unit evaluates the performance of the second model using a difference between the first feature and a second feature obtained by correcting the first feature. The welding is suitable for the welding quality. The appropriateness of the first image means that the first image can be used to determine whether welding is appropriate or not.

2. The determination device according to claim 1, wherein: The processing unit outputs information requesting re-learning or replacement of the second model when the difference is larger than a preset first threshold value.

3. The determination device according to claim 1 or 2, wherein: When it is determined that welding is inappropriate, the third model further outputs a correction amount for welding conditions.

4. The determination device according to claim 1 or 2, wherein: The second model can correct the image and output features for the corrected image.

5. The determination device according to claim 4, wherein: The correction includes rotation, enlargement, reduction or contrast adjustment of the image.

6. The determination device according to claim 4, wherein: The correction includes the removal of defects, damage or weld bead marks.

7. The determination device according to claim 4, wherein: The correction includes an increase or decrease in the width of the weld bead.

8. A determination system comprising: The determination device according to any one of claims 1 to 7; and A learning device enables the second model to learn, The learning device performs a first action and a second action, The first operation is to use a plurality of input images and a plurality of teaching images to allow the second model to learn. The second action is to cause the second model to learn using another input image and a corrected image obtained by inputting the other input image to the second model and correcting an output image output from the second model.

9. A welding system, wherein: have: The determination device according to any one of claims 1 to 7; and The welding device includes an imaging unit for acquiring the first image.

10. A determination method, The first image obtained by photographing the welding part during welding is input to the first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the first image is input to a second model for extracting features related to welding from an image, and the first feature extracted by the second model is input to a third model for determining whether welding is appropriate, thereby determining whether welding shown in the first image is appropriate, and when the first model determines that the first image is inappropriate, outputting information for correcting the shooting conditions of the welding part, evaluating the performance of the second model using a difference between the first feature and a second feature obtained by correcting the first feature, The welding is suitable for the welding quality. The appropriateness of the first image means that the first image can be used to determine whether welding is appropriate or not.

11. A storage medium storing a program for causing a processing unit to execute the following processing: A first image obtained by photographing a welding portion during welding is input to a first model for determining whether the image is appropriate. When the first model determines that the first image is appropriate, the first image is input to a second model for extracting features related to welding from an image, and the first feature extracted by the second model is input to a third model for determining whether welding is appropriate, thereby determining whether welding shown in the first image is appropriate, and when the first model determines that the first image is inappropriate, outputting information for correcting the shooting conditions of the welding part. evaluating the performance of the second model using a difference between the first feature and a second feature obtained by correcting the first feature, The welding is suitable for the welding quality. The appropriateness of the first image means that the first image can be used to determine whether welding is appropriate or not.

Images