Welding systems, welding methods, welding auxiliary devices, computer program products, learning devices, and methods for generating learned models.

Abstract

The welding system includes: a welding device; multiple sensors of different types that detect events accompanying the welding performed by the welding device; and an estimation unit that uses a learned model to estimate the degree of abnormality of the welding performed by the welding device based on multiple detection data generated by the multiple sensors. The learned model is pre-generated using machine learning, taking multiple learning data of events accompanying the welding detected by sensors of the same type as the multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data.

Description

Technical Field

[0001] This invention relates to welding systems, welding methods, welding auxiliary devices, programs, learning devices, and methods for generating learned models. Background Technology

[0002] Patent Document 1 discloses the following technology: a camera captures images of the molten pool, the shape of the weld ripple, and the geometry of the fillet weld; a processor receives the images and communicates with a database that stores images of the molten pool, the weld ripple shape, and the geometry of the fillet weld of a simulated weld, along with images of potential defects associated with the weld in the database; and calculates the total probability that a defect is present at the weld location corresponding to the image captured by the camera, based on the potential defects associated with the database.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-106908 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, welding defects can occur on the surface or internally, but those that can be inspected by camera are mainly limited to those occurring on the surface. Furthermore, in arc welding, the weld area can be obscured by the arc light, making it difficult to accurately inspect weld quality using only camera images.

[0008] Furthermore, the welding sound or the waveform of the current or voltage of the welding power source may become disordered due to poor welding, but this does not necessarily correspond to the occurrence of welding defects. Therefore, it is difficult to accurately check the welding quality by relying solely on their respective measurement data.

[0009] The present invention was made in view of the above-mentioned problems, and its main objective is to provide a welding system, welding method, welding auxiliary device, program, learning device, and method for generating a learned model that can improve the prediction accuracy of welding quality.

[0010] Methods used to solve problems

[0011] To address the aforementioned issues, one aspect of the present invention provides a welding system comprising: a welding apparatus; multiple sensors of different types that detect events accompanying welding performed by the welding apparatus; and an estimation unit that uses a learned model to estimate the degree of abnormality of welding performed by the welding apparatus based on multiple detection data generated by the multiple sensors. The learned model is pre-generated using machine learning, taking multiple learning data of events accompanying welding detected by sensors of the same type as the multiple sensors as input data and labels indicating whether the welding is normal or abnormal as teaching data.

[0012] Furthermore, another aspect of the welding method of the present invention involves using multiple sensors of different types to detect events accompanying welding performed by a welding device; and using a learned model to estimate the degree of abnormality of welding performed by the aforementioned welding device based on multiple detection data generated by the aforementioned multiple sensors. The learned model is pre-generated using multiple learning data of events accompanying welding detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data.

[0013] Furthermore, another aspect of the welding assistance device of the present invention comprises: an acquisition unit that acquires multiple detection data generated by multiple sensors of different types, the multiple sensors detecting events accompanying welding performed by a welding device; and an estimation unit that uses a learned model to estimate the degree of abnormality of welding performed by the welding device based on the multiple detection data generated by the aforementioned multiple sensors, the learned model being pre-generated using multiple learning data of events accompanying welding detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data.

[0014] Furthermore, another aspect of the present invention involves a computer performing the following steps: acquiring multiple detection data generated by multiple sensors of different types, the multiple sensors detecting events accompanying welding performed by a welding apparatus; and using a learned model to estimate the degree of abnormality of the welding performed by the aforementioned welding apparatus based on the multiple detection data generated by the aforementioned multiple sensors, the learned model being pre-generated using machine learning, with multiple learning data of events accompanying welding detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data.

[0015] Furthermore, another aspect of the present invention provides a learning apparatus comprising: an acquisition unit that acquires a learning dataset, the learning dataset including multiple learning data points detected by multiple sensors of different types that accompany welding events and labels indicating whether the welding is normal or abnormal; and a learning unit that generates a learned model using the aforementioned multiple learning data points as input data and the aforementioned labels as teaching data, the learned model being used to estimate the degree of abnormality of the welding based on multiple detection data points detected by sensors of the same type as the aforementioned multiple sensors that accompany welding events.

[0016] Furthermore, another aspect of the present invention provides a method for generating a learned model, which involves obtaining a learning dataset containing multiple learning data points detected by multiple sensors of different types that accompany welding, and labels indicating whether the welding is normal or abnormal; using the aforementioned multiple learning data points as input data and the aforementioned labels as teaching data, a learned model is generated, which is used to estimate the degree of abnormality of the welding based on multiple detection data points detected by sensors of the same type as the aforementioned multiple sensors that accompany welding.

[0017] Invention Effects

[0018] According to the present invention, the accuracy of welding quality prediction can be improved. Attached Figure Description

[0019] Figure 1 This is a diagram showing an example of the structure of a welding system.

[0020] Figure 2 This is a diagram showing an example of welding performed by a welding apparatus.

[0021] Figure 3 It is a diagram used to illustrate the dataset used for learning.

[0022] Figure 4 This is an example of an image obtained by photographing the welded part during welding.

[0023] Figure 5 This is an example of a spectrum diagram representing welding sounds.

[0024] Figure 6 This is a diagram showing an example of the voltage and current of a welding power source.

[0025] Figure 7 This is a graph representing an example of the number of short circuits per unit of time.

[0026] Figure 8 This is a graph representing examples of learned models.

[0027] Figure 9 This is a diagram illustrating the sequence of learning stages.

[0028] Figure 10 This is a diagram illustrating the sequence of inference stages.

[0029] Figure 11 This is a diagram showing another structural example of a welding auxiliary device.

[0030] Figure 12 This is another example of a dataset used for learning.

[0031] Figure 13 This is a diagram representing another example of a learned model.

[0032] Figure 14 This is a diagram representing another sequence of the reasoning stage.

[0033] Figure 15 This is a diagram representing another sequence of learning stages.

[0034] Figure 16 This is a graph showing an example of how the shape of a molten pool changes over time.

[0035] Figure 17 This is a diagram representing another example of a learned model.

[0036] Figure 18 This is a diagram representing another sequence of the reasoning stage. Detailed Implementation

[0037] The following is a reference to the appendix. Figure 1 The embodiments of the present invention will be described below.

[0038] [First Implementation]

[0039] (1) System Overview

[0040] Figure 1 This is a diagram illustrating a structural example of the welding system 100 according to the embodiment. Figure 2 This is a diagram showing a welding example performed by the welding apparatus 3 of the welding system 100.

[0041] like Figure 1 As shown, the welding system 100 includes a welding auxiliary device 1, a camera 21, a microphone 22, a welding device 3, a welding power source 4, a storage device 5, and a learning device 6. The camera 21, microphone 22, welding device 3, welding power source 4, and storage device 5 are communicatively connected to the welding auxiliary device 1.

[0042] The welding auxiliary device 1 is a computer including a CPU, GPU, RAM, ROM, non-volatile memory, and input / output interfaces. The CPU of the welding auxiliary device 1 performs information processing according to a program loaded from ROM or non-volatile memory into RAM.

[0043] The welding auxiliary device 1 includes an acquisition unit 11, a conversion unit 12, and an estimation unit 13. These functional units are implemented by the CPU of the welding auxiliary device 1 executing information processing according to a program.

[0044] The program can be provided via information storage media such as optical discs or memory cards, or via communication networks such as the Internet or LAN.

[0045] The learning device 6 is also the same computer as the welding auxiliary device 1. The learning device 6 has an acquisition unit 61 and a learning unit 62.

[0046] The welding aid device 1 and the learning device 6 can access the storage device 5. The storage device 5 stores the learned model M generated by the learning device 6, which can be read out using the welding aid device 1.

[0047] like Figure 2 As shown, in this embodiment, the welding apparatus 3 is a welding robot that performs arc welding while moving the welding torch 31 through a bevel G formed between the two parts U and L to be welded. A molten pool P is formed in the weld area during arc welding.

[0048] In the illustrated example, the welded components U and L are arranged in the vertical direction (up and down), and the bevel G extends in the horizontal direction (back and forth). However, this is not a limitation; the welded components U and L can also be arranged in the horizontal direction.

[0049] The spacing between the welded components U and L is approximately 3 to 10 mm. Backing materials may or may not be attached to the welded components U and L. The shape of the bevel G is not limited to the V-shape shown in the diagram; it can also be an X-shape, etc.

[0050] Arc welding includes, for example, TIG (Tungsten Inert Gas) welding. It is not limited to this; it can also include MIG (Metal Inert Gas) welding or MAG (Metal Active Gas) welding, etc.

[0051] The welding device 3 performs arc welding while the welding torch 31 is oscillating laterally. For example, when the parts U and L to be welded are arranged vertically and the welding direction is forward, the welding torch 31 is oscillating laterally in the front-lower to rear-up direction in order to suppress the drooping of the molten pool P.

[0052] Camera 21 captures images of the welded portion during arc welding, generating image data. Hereinafter, the image data generated by camera 21 will be referred to as "camera image". Camera 21 is an example of a sensor that detects events accompanying arc welding, and camera image is an example of learning data and detection data.

[0053] Specifically, camera 21 captures images of the molten pool P formed near the tip of welding torch 31. The area captured by camera 21 includes not only the molten pool P, but also the electric arc generated from the tip of welding torch 31 and the welding wire (filler material) being fed towards the molten pool P.

[0054] The camera 21 is positioned in front of or behind the welding torch 31 and moves forward together with the welding torch 31. A bandpass filter, which allows only near-infrared light around 950nm to pass through, is installed in the lens of the camera 2 to suppress the incidence of arc light.

[0055] Camera 21 is a video camera that generates motion images containing multiple still images (frames) in a time series. However, it is not limited to this; camera 21 can also be a still camera that generates multiple still images in a time series by periodically taking pictures.

[0056] Microphone 22 records the welding sounds during arc welding to generate sound data. Hereinafter, the welding sound data generated by microphone 22 will be simply referred to as "welding sound". Microphone 22 is an example of a sensor that detects events accompanying arc welding, and welding sound is an example of learning data and detection data.

[0057] Welding power source 4 (reference) Figure 1 The welding power source 4 supplies power to the welding apparatus 3 for arc welding. The output characteristics of the welding power source 4 may include, for example, constant current or constant voltage characteristics. The welding power source 4 includes a voltmeter and an ammeter to measure the voltage and current during arc welding.

[0058] Hereinafter, the voltage and current data measured by the welding power source 4 during arc welding will be simply referred to as "voltage" and "current". The voltmeter and ammeter included in the welding power source 4 are examples of sensors used to detect events accompanying arc welding, and the voltage and current are examples of data for learning and detection.

[0059] In this embodiment, camera 21, microphone 22 and welding power source 4 are examples of sensors, but the sensors are not limited to these examples as long as they can detect things that accompany arc welding.

[0060] (2) Learning stage

[0061] The following describes the learning stage of the learned model generation method implemented in the learning device 6 as an embodiment. In this embodiment, so-called multimodal learning is performed using multiple types of data as input data.

[0062] Figure 3 This is a diagram illustrating the learning dataset used in the learning phase. The learning dataset contains camera images of the weld during arc welding, a spectrogram of the welding sound, and labels indicating normal / abnormal conditions. The camera images and the spectrogram of the welding sound are examples of several learning datasets used as input data. The labels are used as teaching data.

[0063] Figure 4 This is a diagram illustrating an example of a camera image. The camera image includes the molten pool, the electric arc, and the welding wire. The camera image is generated by the camera 21 capturing images at a rate of, for example, approximately 100 fps (frames per second). In this diagram, for illustration purposes, dashed lines or circular markings are used to represent the left end, right end, and rear end of the molten pool, the center of the electric arc, and the front end of the welding wire. These will be treated as feature points in the second embodiment described later.

[0064] Figure 5 This is an example of a spectrogram representing a welding sound. A spectrogram is a graph that represents the welding sound using three dimensions: time, frequency, and intensity, through a continuous Fourier transform over time. In this graph, the horizontal axis represents time, the vertical axis represents frequency, and the color (or brightness) represents intensity. The welding sound is generated by recording with microphone 22 at a sampling rate of, for example, approximately 44.1 kHz.

[0065] like Figure 3 As shown, a spectrogram is created by extracting welding sound data from a specified period of time after the camera image was captured, and then correlated with the camera image. For example, if the camera image was captured at second a, the welding sound data from 1 second between a-1 and a is extracted to create a spectrogram, and then correlated with the camera image captured at second a.

[0066] In this embodiment, camera images and welding sounds are used as input data (i.e., as examples of learning data and detection data), but the input data is not limited to these. The input data may also include two or more of the following: camera images, welding sounds, the voltage of the welding power source, and the current of the welding power source; however, it is preferred to include at least one of the camera images and welding sounds. Furthermore, when using the voltage or current of the welding power source, a spectrum diagram is generated in the same manner as for the welding sounds.

[0067] The labels indicate whether the welding is normal or abnormal. For example, if an abnormality such as burn-through is seen in a camera image of the weld during arc welding, the camera image is labeled "abnormal." Alternatively, the label can be assigned to periods with abnormal welding sounds, or to periods where there are disturbances in the voltage or current waveforms (an increase in the number of short circuits, a deviation from the average value).

[0068] Welding anomalies can also be identified through post-arc welding inspections. For example, camera images obtained during the welding process corresponding to the location of the welding anomaly can be labeled as "abnormal," and the location of the anomaly can be determined based on information such as visual inspection, the timing of welding condition switching, and non-destructive testing such as ultrasonic testing or X-rays.

[0069] In addition to defects on the surface or inside the weld (pits, porosity, slag entrapment, poor fusion, etc.), welding abnormalities also include burn-through of the molten pool and poor shielding gas.

[0070] In addition, in this embodiment, the camera images and welding sounds generated by the sensors such as the camera 21 and microphone 22 included in the welding system 100 are used as learning data. However, it is not limited to this. Detection data generated by other sensors of the same type as the camera 21 and microphone 22 can also be used as learning data.

[0071] Figure 6 This is a graph illustrating an example of the voltage and current of welding power source 4. In this graph, the horizontal axis represents time, and the vertical axis represents the magnitude of the voltage and current. Solid lines in the graph represent voltage, and dashed lines represent current. The voltage and current are obtained from welding power source 4 through analog-to-digital conversion at, for example, a sampling rate of 20 kHz (0.05 ms interval).

[0072] When using welding wire (filler material), observe the current / voltage waveform of droplet transfer, which is called short-circuit transfer. That is, if the droplet generated at the end of the welding wire contacts the molten pool (short circuit), a waveform change occurs with a sudden drop in potential difference and a sudden increase in current.

[0073] Figure 7 This is a graph illustrating an example of the number of short circuits per unit of time. In this graph, the horizontal axis represents time, and the vertical axis represents the number of short circuits per unit of time (T seconds). Additionally, the above... Figure 6 The width of the horizontal axis is approximately equivalent to a unit of time (T seconds).

[0074] Using this result, the short-circuit increase interval, where the number of short circuits increases, is defined as an abnormal interval. Furthermore, there are also intervals where defects occur even though the number of short circuits has not increased, and where not all defects in the welded parts can be detected solely by voltage or current waveforms. Therefore, the short-circuit increase interval can also be treated as an abnormal voltage or current quality interval in the second embodiment described later.

[0075] Figure 8 This is a diagram representing an example of a learned model M generated from the learning phase. The learned model M is, for example, a Convolutional Neural Network (CNN), which includes periodically arranged convolutional layers, normalization layers, pooling layers, and an output layer set in the final stage.

[0076] Specifically, the learned model M is a multimodal CNN comprising multiple networks M1 and M2. The multiple networks M1 and M2 each include periodically arranged convolutional layers, normalization layers, and pooling layers, which are combined with the output layer set in the final segment.

[0077] In this embodiment, the learned model M extracts features by convolving the camera image with the first network M1 and extracts features by convolving the spectrum of the welding sound with the second network M2. The features of the two networks are combined immediately before the output layer to estimate the welding anomaly degree (the probability of welding anomaly).

[0078] The output layer includes elements corresponding to the degree of welding anomalies. These elements use functions such as the sigmoid function or the softmax function to output values ​​between 0 and 1. The output value of the output layer is used as the degree of welding anomalies; for example, values ​​closer to 0 indicate a more normal weld, while values ​​closer to 1 indicate a more abnormal weld.

[0079] Figure 9 This is a diagram illustrating the sequence of learning stages implemented in learning device 6. Learning device 6 executes the information processing shown in this diagram according to the program.

[0080] First, the learning device 6 acquires a learning dataset containing camera images, a spectrogram of welding sounds, and normal / abnormal labels (S11: as processing by the acquisition unit 61).

[0081] Next, the learning device 6 inputs the camera image and the spectrum of the welding sound as input data into the model (the unlearned model M), performs calculations using the model, and outputs the welding anomaly degree from the model (S12~S14: processing by the learning unit 62).

[0082] Next, the learning device 6 calculates the error between the welding anomaly degree, which is the output data from the model, and the normal / abnormal label, which is the teaching data, and performs backpropagation calculation to reduce the error (S15, S16: processing by the learning unit 62).

[0083] By repeatedly performing the above processing, a learned model M is generated to estimate the degree of welding anomaly based on the spectrum of camera images and welding sounds. The learned model M generated by the learning device 6 is stored in the storage device 5 and used in the inference stage described below.

[0084] (3) Inference stage

[0085] The following describes the inference stage implemented in welding auxiliary device 1. Figure 10 This diagram illustrates the sequence of the deduction phase. Welding auxiliary device 1 executes the information processing shown in this diagram according to the procedure. The deduction phase is performed during arc welding by welding device 3.

[0086] First, the welding auxiliary device 1 acquires a camera image of the welded part in arc welding captured by the camera 21 and a welding sound in arc welding recorded by the microphone 22 (S21: as processing of the acquisition unit 11).

[0087] Next, the welding auxiliary device 1 generates a spectrum of the obtained welding sound (S22: processing as a transformation unit 12).

[0088] Specifically, the welding auxiliary device 1 sequentially acquires multiple still images (frames) containing the time sequence of the motion image generated by the camera 21 as camera images, and cuts out the welding sound during a period of time (e.g., 1 second) back from the shooting time of the camera image, transforms it into a spectrum diagram, and generates a group of spectrum diagrams of the camera image and the welding sound.

[0089] Next, the welding assistance device 1 uses the learned model M generated and stored in the storage device 5 during the above-mentioned learning phase to estimate the degree of welding abnormality based on the camera image and the spectrum of welding sound (S23: processing as estimation unit 13).

[0090] Specifically, the welding auxiliary device 1 inputs the camera image and the spectrum of the welding sound as input data into the learned model M, performs calculations using the learned model M, and outputs the welding anomaly degree from the learned model M.

[0091] Next, if the welding abnormality level is above the threshold (S24: Yes), the welding auxiliary device 1 notifies the welding device 3 of the welding abnormality (S25). If the welding device 3 receives the notification from the welding auxiliary device 1, it performs a prescribed action such as slowing down or stopping the movement of the welding torch 31.

[0092] (4) Effect

[0093] According to the implementation method described above, since the learned model M is used to estimate the degree of welding abnormality based on the camera image and welding sound generated during arc welding, the welding quality can be checked online with high precision.

[0094] Furthermore, according to the implementation method, since the training dataset (see reference) is generated... Figure 3 When using teaching data, it is sufficient to associate the labels representing normal / abnormal data with learning data such as camera images, thus reducing the cost of producing teaching data.

[0095] Furthermore, according to the implementation method, by converting the welding audio and video into a spectrogram, it can be processed as input data for a CNN, just like camera images. By convolving the spectrogram, the relationship between time, frequency, and intensity can be learned.

[0096] Furthermore, according to the implementation method, the learned model M (refer to...) Figure 8 Multimodal CNNs can complementarily utilize the feature quantities extracted by each network M1 and M2. Therefore, welding anomalies can be detected earlier compared to relying solely on images.

[0097] Furthermore, the combination of multiple networks M1 and M2 can also be performed in earlier layers. In addition, networks for voltage or current spectrograms can be added to create a 3- or 4-column multimodal CNN.

[0098] [Second Implementation]

[0099] The second embodiment will be described below. For structures that are repeated in the above embodiments, detailed descriptions may be omitted by using the same reference numerals.

[0100] Figure 11 This is a diagram showing a structural example of the welding auxiliary device 1B according to the second embodiment. In addition to the acquisition unit 11, the transformation unit 12, and the estimation unit 13, the welding auxiliary device 1B also includes a similarity calculation unit 14 and a reliability determination unit 15.

[0101] In this embodiment, the estimation unit 13 estimates not only the degree of welding anomaly but also the feature quantities of the detection data. The similarity calculation unit 14 calculates the similarity between the feature quantities extracted from the learning data and the feature quantities estimated by the estimation unit 13. The reliability determination unit 15 determines the reliability of the degree of welding anomaly estimated by the estimation unit 13 based on the calculated similarity.

[0102] Figure 12This is a diagram illustrating an example of a training dataset used during the learning phase. In addition to camera images, spectrograms of welding sounds, and labels indicating normal / abnormal conditions, the training dataset includes the locations of feature points in the camera images and the quality aberrations in the welding sounds. The locations of feature points and quality aberrations are examples of feature quantities used as teaching data.

[0103] The feature points of the camera image are as described above. Figure 4 As shown, this includes five points: the left end of the molten pool, the right end of the molten pool, the rear end of the molten pool, the center of the arc, and the welding wire. The left and right ends of the molten pool are represented only by x-coordinates, the rear end of the molten pool by only y-coordinates, and the center of the arc and the welding wire by both x and y coordinates. In other words, the five characteristic points are represented by seven variables.

[0104] An abnormal quality range for welding sounds is defined as a period in the waveform where disturbances or other abnormalities are visible. For example, if it cannot be determined to be a welding abnormality, but disturbances or other abnormalities are visible in the waveform, indicating an inherent quality issue in the welding sound itself, an abnormal quality range is assigned. Alternatively, an abnormal quality range can be partially assigned in the frequency spectrum.

[0105] Figure 13 This is a diagram representing an example of a learned model M generated during the learning phase. The learned model M includes multiple output layers that combine multiple networks M1 and M2 respectively. Output layer 1 corresponds to the welding anomaly degree, output layer 2 corresponds to the location of feature points in the camera image, and output layer 3 corresponds to the quality anomaly range of the welding sound.

[0106] In this embodiment, the learned model M extracts features by convolving the camera image with the first network M1 and extracts features by convolving the spectrum of the welding sound with the second network M2. The features of the two are combined immediately before the output layer to estimate the welding anomaly degree, the position of the feature points of the camera image, and the quality anomaly range of the welding sound.

[0107] During the learning phase, not only is the error between the welding anomaly degree output from output layer 1 and the normal / abnormal label used as teaching data calculated, but also the error between the position of the feature point output from output layer 2 and the position of the feature point used as teaching data, and the error between the quality anomaly interval output from output layer 3 and the quality anomaly interval used as teaching data are calculated. Error backpropagation calculations are performed to reduce these errors.

[0108] Figure 14 This diagram illustrates an example of the sequence of the deduction stages implemented in welding auxiliary device 1. For steps that are repeated in the above embodiments, detailed descriptions may be omitted by assigning the same reference numerals.

[0109] First, the welding auxiliary device 1 acquires a camera image of the welded part in arc welding captured by the camera 21 and a welding sound in arc welding recorded by the microphone 22 (S21: as processing of the acquisition unit 11), and generates a spectrum diagram of the acquired welding sound (S22: as processing of the transformation unit 12).

[0110] Next, the welding assist device 1 uses the learned model M to estimate the welding anomaly degree, the location of feature points in the camera image, and the quality anomaly range of the welding sound based on the spectrum of the camera image and the welding sound (S33: processing as an estimation unit 13).

[0111] Specifically, the welding auxiliary device 1 inputs the camera image and the spectrum of the welding sound as input data to the learned model M, performs calculations with the help of the learned model M, and outputs the welding anomaly degree, the position of the feature points of the camera image, and the quality anomaly range of the welding sound from the learned model M.

[0112] Next, the welding assist device 1 calculates the similarity between the feature quantity extracted from the learning data in advance and the feature quantity estimated in S33 (S34: as processing of the similarity calculation unit 14), and determines the reliability of the welding anomaly estimated in S33 based on the calculated similarity (S35: as processing of the reliability determination unit 15).

[0113] In the calculation of similarity, methods such as measuring similarity in vector space (cosine similarity) are used. For example, the cosine similarity between the vector d of the presumed data and the vector q of the learning data is expressed by the following formula (1).

[0114] [Formula 1]

[0115]

[0116] Here, the vector d of the estimated data represents the estimated position coordinates of the feature points in the camera image. Specifically, the vector d of the estimated data is represented by (arc center x, arc center y, welding wire x, welding wire y, molten pool - left x, molten pool - right x, molten pool - rear end y).

[0117] The vector q of the learning data represents the position coordinates of feature points randomly extracted from the learning dataset. Specifically, the vector q of the learning data is also represented by (arc center x, arc center y, welding wire x, welding wire y, molten pool - left x, molten pool - right x, molten pool - rear end y).

[0118] If the cosine similarity value obtained from the above formula (1) is close to 1, the distribution of the presumed data is close to that of the training data, and the reliability of the welding anomaly is high. On the other hand, if the cosine similarity value is, for example, less than 0.5, there is a possibility that the presumed data is outside the distribution of the training data, and the reliability of the welding anomaly is low.

[0119] Next, if the reliability of the welding abnormality is above the threshold and the welding abnormality is above the threshold (S35: Yes, and S24: Yes), the welding auxiliary device 1 notifies the welding device 3 of the welding abnormality (S25).

[0120] Specifically, assuming similarity equals reliability, in S35, the welding auxiliary device 1 determines whether the similarity is above a threshold. When using the cosine similarity of the above formula (1), the threshold is, for example, 0.5.

[0121] According to the second embodiment described above, since the similarity between the feature quantity extracted from the learning data and the feature quantity inferred from the learned model M is calculated, and the reliability of the welding anomaly degree is determined based on the calculated similarity, the welding quality can be checked online with higher accuracy.

[0122] Furthermore, in this embodiment, both the location of feature points in the camera image and the quality anomaly range of the welding sound are learned / estimated, but it is not limited to these; feature quantities of either one can also be learned / estimated. Additionally, when the voltage or current of the welding power source is used in the input data, the quality anomaly range of the voltage or current can also be learned / estimated.

[0123] [Third Implementation]

[0124] The third embodiment will now be described. For structures that are repeated in the embodiments described above, detailed descriptions may be omitted by using the same reference numerals.

[0125] In this embodiment, multimodal learning is performed, which uses time-series data of the weld pool shape measured from camera images to estimate the weld quality. Therefore, even when it is difficult to make a normal / abnormal judgment using camera images, normal / abnormal conditions can be determined with high accuracy by using time-series data of the weld pool shape.

[0126] (1) Learning stage

[0127] Explain the learning stages. Figure 15 This is a diagram illustrating the order in which learning data is generated during the learning phase of this embodiment.

[0128] First, the learning device 6 acquires camera images and current and voltage waveforms measured during arc welding for learning performed by the welding device 3 (S41).

[0129] Next, the learning device 6 uses the learned model for feature extraction to estimate the shape of the molten pool based on the camera images (S42). Specifically, the learning device 6 estimates the shape of the molten pool sequentially based on multiple time-series camera images, and generates time-series data of the molten pool shape.

[0130] The learned model for feature extraction is a learned model used to estimate the locations of feature points in a camera image. The learned model for feature extraction is generated using machine learning, with the camera image as input data and the locations of the feature points as teaching data. The learned model for feature extraction is the same as the network included in the learned model described in the second embodiment above, which outputs the locations of the feature points (see [reference]). Figure 13 ).

[0131] The feature points of the camera image are as described above. Figure 4 As shown, the feature points include the left end of the molten pool, the right end of the molten pool, the rear end of the molten pool, the center of the arc, and the front end of the welding wire. Furthermore, the feature points of the camera image may also include the front end of the molten pool.

[0132] The shape of the molten pool is, for example, the molten pool width, which represents the width of the molten pool in the left-right direction. The molten pool width is represented by the distance between the positions of the left and right ends of the molten pool, which is estimated by the learned model used for feature extraction.

[0133] Furthermore, the shape of the molten pool can also be, for example, a molten pool advance amount, which represents the amount by which the tip of the molten pool advances relative to the lead of the welding wire. The molten pool advance amount is represented by the distance between the position of the tip of the welding wire and the position of the tip of the molten pool, as estimated by the learned model used for feature extraction.

[0134] Next, the learning device 6 establishes a correlation between the abnormal location and the time-series data of the molten pool shape and the current and voltage waveforms (S43). Specifically, the learning device 6 divides the time-series data into intervals of a predetermined time (e.g., every 1 second) and assigns a normal / abnormal label to each interval. In this way, a learning dataset is created.

[0135] Figure 16 This is a graph showing the change in the shape of the molten pool over time. The horizontal axis represents time, and the vertical axis represents the width of the molten pool. In the experiment, poor shielding gas was intentionally created by blowing air onto the weld area with a fan, aiming to produce porosity defects. "On" and "Off" in the graph indicate the fan's ON / OFF state.

[0136] As shown in the figure, during the period when the fan is on, disorder (the part surrounded by the ellipse with double-dotted lines) can be seen in the width of the molten pool. That is, it is confirmed that disorder occurs in the shape of the molten pool along with welding abnormalities.

[0137] Figure 17 This is a diagram illustrating an example of a learned model generated by the learning phase of this embodiment. The learned model includes convolutional layers, RNNs (Recurrent Neural Networks), pooling layers, and an output layer.

[0138] Specifically, the learned model is a multimodal RNN consisting of multiple networks M3 and M4. Each of the multiple networks M3 and M4 contains convolutional layers, RNNs, and pooling layers, which are combined with the output layer set in the final segment.

[0139] The learned model extracts features from the time series data of current and voltage waveforms by the first network M3 and features from the time series data of molten pool shape by the second network M4. The features of the two networks are combined immediately before the output layer to estimate the welding anomaly degree.

[0140] In this embodiment, time series data of current and voltage waveforms and time series data of molten pool shape are used as input data (i.e., as examples of learning data and detection data), but the input data is not limited to these.

[0141] (2) Inference stage

[0142] The reasoning stage is explained. Figure 18 This diagram illustrates an example of the sequence of the deduction phases in this embodiment. The welding auxiliary device 1 executes the information processing shown in this diagram according to the program. The deduction phase is performed during the arc welding process by the welding device 3.

[0143] First, the welding auxiliary device 1 acquires a camera image of the welded part in arc welding captured by the camera 21 and a current and voltage waveform in arc welding measured by the welding power source 4 (S51: as processing of the acquisition unit 11).

[0144] Next, the welding auxiliary device 1 uses the learned model for feature extraction to estimate the shape of the molten pool based on the camera image and generates time series data of the molten pool shape (S52). This process is the same as S42 in the learning phase.

[0145] Next, welding auxiliary device 1 uses the learned model for anomaly detection generated during the learning phase (refer to...). Figure 17 The degree of welding abnormality is estimated based on the time series data of the molten pool shape and the time series data of the current and voltage waveforms (S53: processing as estimation unit 13).

[0146] Specifically, the welding auxiliary device 1 inputs the time series data of the molten pool shape and the time series data of the current and voltage waveforms into the learned model as input data, performs calculations using the learned model, and outputs the welding anomaly degree from the learned model.

[0147] Next, if the welding abnormality level is above the threshold (S24: Yes), the welding auxiliary device 1 notifies the welding device 3 of the welding abnormality (S25). If the welding device 3 receives the notification from the welding auxiliary device 1, it performs a prescribed action such as slowing down or stopping the movement of the welding torch 31.

[0148] According to the implementation method described above, even when it is difficult to make a normal / abnormal judgment using camera images, welding quality can be checked online with high precision by using time series data of the molten pool shape.

[0149] The embodiments of the present invention have been described above, but the present invention is not limited to the embodiments described above, and various modifications can certainly be made by those skilled in the art.

[0150] The above is with reference to the appendix. Figure 1 Various embodiments have been described, but the present invention is certainly not limited to such examples. It will be apparent to those skilled in the art that various modifications or alterations will arise within the scope of the claims, and that they also fall within the scope of the present invention. Furthermore, the constituent elements of the above embodiments can be arbitrarily combined without departing from the spirit of the invention.

[0151] Furthermore, the contents of Japanese Patent Application No. 2020-174445, filed on October 16, 2020, are incorporated herein by reference.

[0152] Explanation of reference numerals in the attached figures

[0153] 1 Welding auxiliary device; 11 Acquisition unit; 12 Transformation unit; 13 Estimation unit; 14 Similarity calculation unit; 15 Reliability determination unit; 21 Camera; 22 Microphone; 3 Welding apparatus; 31 Welding torch; 4 Welding power source; 5 Storage device; 6 Learning device; 61 Acquisition unit; 62 Learning unit; 100 Welding system; M Learned model; U Welded part; L Welded part; G Groove; P Molten pool

Claims

1. A welding system, characterized in that, have: Welding equipment; Multiple sensors of different types detect welding processes performed by the aforementioned welding apparatus; and The estimation unit uses a learned model to estimate the degree of abnormality of the welding performed by the aforementioned welding device based on multiple detection data generated by the aforementioned multiple sensors. The learned model is pre-generated using machine learning, with multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learned model is also generated using at least one feature of the aforementioned multiple learning data as teaching data; The aforementioned estimation unit also uses the aforementioned learned model to estimate the feature quantity of at least one of the aforementioned multiple detection data.

2. The welding system as claimed in claim 1, characterized in that, The aforementioned multiple detection data include two or more of the following: images obtained by photographing the welded part during welding, welding sounds, voltage of the welding power source, and current of the welding power source, and include at least one of the aforementioned images and welding sounds.

3. The welding system as described in claim 1 or 2, characterized in that, One of the aforementioned data points is the welding sound, the voltage of the welding power source, or the current of the welding power source. It also includes a conversion unit that generates a spectrum diagram that represents the aforementioned welding sound, the voltage of the welding power source, or the welding power source in three dimensions: time, frequency, and intensity. The aforementioned estimation unit inputs the aforementioned spectrogram into the aforementioned learned model.

4. The welding system as described in claim 1 or 2, characterized in that, One of the aforementioned detection data is an image obtained by photographing the welded part during welding; The aforementioned estimation unit estimates the feature points in the aforementioned image as the aforementioned feature quantity.

5. The welding system as described in claim 1 or 2, characterized in that, One of the aforementioned data points is the welding sound, the voltage of the welding power source, or the current of the welding power source. The aforementioned estimation section estimates the aforementioned abnormal range of welding sound, welding power source voltage, or welding power source current as the aforementioned characteristic quantity.

6. The welding system as described in claim 1 or 2, characterized in that, It also has: The similarity calculation unit calculates the similarity between the features extracted from the aforementioned learning data and the features inferred by the aforementioned inference unit; and The reliability determination unit determines the reliability of the welding anomaly degree estimated by the aforementioned estimation unit based on the aforementioned similarity.

7. A welding system, characterized in that, have: Welding equipment; Multiple sensors of different types detect welding processes performed by the aforementioned welding apparatus; and The estimation unit uses a first learned model to estimate the degree of abnormality of the welding performed by the aforementioned welding device based on multiple detection data generated by the aforementioned multiple sensors. The first learned model is pre-generated using machine learning, with multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learning data and the aforementioned detection data are time series data; The aforementioned estimation unit uses a second learned model to estimate the feature quantity in at least one of the aforementioned plurality of detection data. The second learned model is pre-generated using machine learning with at least one of the aforementioned plurality of learning data as input data and the feature quantity in the aforementioned at least one learning data as teaching data. The aforementioned estimation unit inputs the time series data of the indicators based on the aforementioned feature quantities into the aforementioned first learned model.

8. The welding system as claimed in claim 7, characterized in that, The first learned model mentioned above includes a recurrent neural network.

9. The welding system as described in claim 7 or 8, characterized in that, The time series data of the aforementioned indicators are time series data of indicators representing the shape of the molten pool in the image obtained by photographing the welded part during welding.

10. A welding method, characterized in that, Multiple sensors of different types are used to detect events accompanying welding performed by the welding device; Using a learned model, the degree of abnormality of welding performed by the aforementioned welding device is estimated based on multiple detection data generated by the aforementioned multiple sensors. The learned model is pre-generated with the help of machine learning, taking multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learned model is also generated using at least one feature of the aforementioned multiple learning data as teaching data; The aforementioned inference also uses the aforementioned learned model to infer the feature quantity of at least one of the aforementioned multiple detection data.

11. A welding method, characterized in that, Multiple sensors of different types are used to detect events accompanying welding performed by the welding device; Using the first learned model, the degree of abnormality of welding performed by the aforementioned welding device is estimated based on multiple detection data generated by the aforementioned multiple sensors. The first learned model is pre-generated with the help of machine learning, taking multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learning data and the aforementioned detection data are time series data; The aforementioned assumption uses a second learned model to estimate the feature quantity in at least one of the aforementioned plurality of detection data. The second learned model is pre-generated using machine learning with at least one of the aforementioned plurality of learning data as input data and the feature quantity in the aforementioned at least one learning data as teaching data. The aforementioned assumption is that time-series data based on the aforementioned feature quantities will be input into the aforementioned first learned model.

12. A welding auxiliary device, characterized in that, have: The acquisition unit acquires multiple detection data generated by multiple sensors of different types, which detect welding activities performed by the welding device; as well as The estimation unit uses a learned model to estimate the degree of abnormality of the welding performed by the aforementioned welding device based on multiple detection data generated by the aforementioned multiple sensors. The learned model is pre-generated using machine learning, with multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learned model is also generated using at least one feature of the aforementioned multiple learning data as teaching data; The aforementioned estimation unit also uses the aforementioned learned model to estimate the feature quantity of at least one of the aforementioned multiple detection data.

13. A welding auxiliary device, characterized in that, have: The acquisition unit acquires multiple detection data generated by multiple sensors of different types, which detect welding activities performed by the welding device; as well as The estimation unit uses a first learned model to estimate the degree of abnormality of the welding performed by the aforementioned welding device based on multiple detection data generated by the aforementioned multiple sensors. The first learned model is pre-generated using machine learning, with multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learning data and the aforementioned detection data are time series data; The aforementioned estimation unit uses a second learned model to estimate the feature quantity in at least one of the aforementioned plurality of detection data. The second learned model is pre-generated using machine learning with at least one of the aforementioned plurality of learning data as input data and the feature quantity in the aforementioned at least one learning data as teaching data. The aforementioned estimation unit inputs the time series data of the indicators based on the aforementioned feature quantities into the aforementioned first learned model.

14. A computer program product, characterized in that, To make the computer perform: Acquire multiple detection data generated by multiple sensors of different types, the multiple sensors detecting events accompanying welding performed by the welding apparatus; and Using a learned model, the degree of abnormality of welding performed by the aforementioned welding device is estimated based on multiple detection data generated by the aforementioned multiple sensors. The learned model is pre-generated with the help of machine learning, taking multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learned model is also generated using at least one feature of the aforementioned multiple learning data as teaching data; The aforementioned inference also uses the aforementioned learned model to infer the feature quantity of at least one of the aforementioned multiple detection data.

15. A computer program product, characterized in that, To make the computer perform: Acquire multiple detection data generated by multiple sensors of different types, the multiple sensors detecting events accompanying welding performed by the welding apparatus; and Using the first learned model, the degree of abnormality of welding performed by the aforementioned welding device is estimated based on multiple detection data generated by the aforementioned multiple sensors. The first learned model is pre-generated with the help of machine learning, taking multiple learning data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors as input data, and labels indicating whether the welding is normal or abnormal as teaching data. The aforementioned learning data and the aforementioned detection data are time series data; The aforementioned assumption uses a second learned model to estimate the feature quantity in at least one of the aforementioned plurality of detection data. The second learned model is pre-generated using machine learning with at least one of the aforementioned plurality of learning data as input data and the feature quantity in the aforementioned at least one learning data as teaching data. The aforementioned assumption is that time-series data based on the aforementioned feature quantities will be input into the aforementioned first learned model.

16. A learning device, characterized in that, have: The acquisition department acquires a learning dataset, which contains multiple learning data points detected by multiple sensors of different types that accompany welding events, and labels indicating whether the welding is normal or abnormal. as well as The learning department uses the aforementioned multiple learning data as input data and the aforementioned tags as teaching data to generate a learned model. The learned model is used to estimate the degree of abnormality of welding based on multiple detection data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors. The aforementioned learning unit also uses the feature value of at least one of the aforementioned learning data as teaching data to generate the aforementioned learned model for further estimating the feature value of at least one of the aforementioned detection data.

17. A method for generating a learned model, characterized in that, Obtain a learning dataset, which contains multiple learning data points detected by multiple sensors of different types that accompany welding events and labels indicating whether the welding is normal or abnormal; Using the aforementioned multiple learning data as input data and the aforementioned labels as teaching data, a learned model is generated. The learned model is used to estimate the degree of abnormality of welding based on multiple detection data of welding-related events detected by sensors of the same type as the aforementioned multiple sensors. The aforementioned generation also uses the feature value of at least one of the aforementioned learning data as teaching data to generate the aforementioned learned model for further estimating the feature value of at least one of the aforementioned detection data.

Images