Information processing apparatus, control apparatus, information processing method, control method, and storage medium

Abstract

To appropriately control welding by allowing a feature point extraction model to appropriately learn events of various feature points.SOLUTION: According to an embodiment, an information processing apparatus includes a learning unit that learns a feature point extraction model that inputs a welding image and outputs position information of a feature point, an evaluation value indicating reliability of the position information, and event information, using training data including one or more welding images obtained by imaging a welding target, a true value of position information of one or a plurality of feature points in the welding image, and event information indicating presence or absence of occurrence of an event related to the welding as a first feature point of which occurrence of an event is unspecified and an extraction number is single among the one or the plurality of feature points.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing device, a control device, an information processing method, a control method, and a program. [Background technology]

[0002] There is known a technology for detecting feature points from a welding image, which is an image of an object to be welded, and controlling welding using the positions of the detected feature points. In detecting feature points from an image, for example, a feature point extraction model is used that is trained using training data including a learning image and true values ​​of the positions of feature points in the image (i.e., ground truth data).

[0003] As a learning method that reduces the load for acquiring training data, for example, a method that uses images obtained during operation of a welding system as training data can be considered. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-182966 Summary of the Invention [Problem to be solved by the invention]

[0005] However, since there are various types of feature point events, it is difficult with conventional technology to train a feature point extraction model to learn the occurrence of all feature point events and to control welding using the positions of feature points extracted using such a feature point extraction model. [Means for solving the problem]

[0006] An information processing device according to an embodiment includes a learning unit that uses training data including one or more welding images of an object to be welded, true values ​​of position information of one or more feature points in the welding images, and event information indicating whether or not an event related to the welding has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of which is single, to input the welding images and learn a feature point extraction model that outputs the position information of the feature points, an evaluation value indicating the reliability of the position information, and the event information. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram illustrating the configuration of a welding system including an information processing device according to a first embodiment. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of the information processing apparatus according to the first embodiment. [Figure 3] FIG. 3 is a flowchart showing an example of the learning process according to the first embodiment. [Figure 4] FIG. 4 is a diagram for explaining an example of feature points in the first embodiment. [Figure 5] FIG. 5 is a diagram for explaining an example of feature points in the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of feature points in the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of information collection processing according to the first embodiment. [Figure 8] FIG. 8 is a flowchart showing an example of the procedure of the learning process for the feature point extraction model in the first embodiment when the extraction target is an electrode or a wire. [Figure 9] FIG. 9 is a diagram showing an example of training data when the extraction target is an electrode or a wire in the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining an example of the learning process in the first embodiment when the extraction target is an electrode or a wire. [Figure 11]FIG. 11 is a flowchart illustrating an example of a procedure of a welding control process according to the first embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of the procedure of the inference process when the extraction target is an electrode or a wire in the first embodiment. [Figure 13] FIG. 13 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode or a wire in the first embodiment. [Figure 14] FIG. 14 is a diagram showing an example of an image of a molten pool when the wire insertion position is appropriate and the state of the wire insertion position in the first embodiment. [Figure 15] FIG. 15 is a diagram showing an example of the state of the wire insertion position in the first embodiment. [Figure 16] FIG. 16 is a diagram showing an example of a molten pool image when the wire insertion position is high in the first embodiment. [Figure 17] FIG. 17 is a diagram showing an example of a state in which a digging defect occurs in the first embodiment. [Figure 18] FIG. 18 is a diagram showing an example of a molten pool image when an indentation defect occurs in the first embodiment. [Figure 19] FIG. 19 is a diagram showing an example of a state in which burn-through occurs and an example of a molten pool image in the first embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of the procedure of the learning process in the first embodiment when the extraction targets are the electrode, the wire, and the droplets when they fall. [Figure 21] FIG. 21 is a diagram showing an example of training data in the first embodiment when the extraction targets are an electrode, a wire, and a droplet of falling droplets. [Figure 22] FIG. 22 is a diagram for explaining an example of the learning process in the first embodiment when the extraction targets are electrodes, wires, and droplets. [Figure 23] FIG. 23 is a flowchart showing an example of the procedure of the inference process when the extraction target is an electrode, a wire, or a droplet in the first embodiment. [Figure 24] FIG. 24 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode, a wire, or a droplet in the first embodiment. [Figure 25] FIG. 25 is a diagram showing an example of an image of a molten pool with slag in the first embodiment. [Figure 26] FIG. 26 is a diagram showing an example of repair work when slag is identified in the first embodiment. [Figure 27] FIG. 27 is a diagram showing an example of a molten pool image in the normal state, in the state where foreign matter is attached to the electrode 21a, and when foreign matter is attached to the electrode 21a in the first embodiment. [Figure 28] FIG. 28 is a diagram showing an example of an image of a cross section of a molten pool or a weld bead relating to an abnormality in the weld bead in the first embodiment. [Figure 29] FIG. 29 is a flowchart showing an example of the procedure of the learning process in the first embodiment when the extraction targets are an electrode, a wire, and multiple slugs. [Figure 30] FIG. 30 is a diagram showing an example of training data when the extraction targets are an electrode, a wire, and a slug in the first embodiment. [Figure 31] FIG. 31 is a diagram showing an example of a welding image in which a plurality of slags exist in the first embodiment. [Figure 32] FIG. 32 is a diagram for explaining an example of the learning process in the first embodiment when the extraction targets are electrodes, wires, and slags. [Figure 33] FIG. 33 is a flowchart showing an example of the procedure of the inference process when the extraction targets are an electrode, a wire, and multiple slugs in the first embodiment. [Figure 34] FIG. 34 is a diagram showing an example of input information, a processing example, and output information when the extraction targets are an electrode, a wire, and a plurality of slugs in the first embodiment. [Figure 35] FIG. 35 is a flowchart showing an example of the procedure of the learning process in the second embodiment when the extraction targets are an electrode, a wire, and a plurality of slugs. [Figure 36] FIG. 36 is a diagram showing an example of training data 1212 in the case where the extraction target is an electrode, a wire, or a slug in the second embodiment. [Figure 37] FIG. 37 is a diagram for explaining an example of the learning process in the second embodiment when the extraction targets are electrodes, wires, and slags. [Figure 38] FIG. 38 is a flowchart showing an example of the procedure of the inference process when the extraction targets are an electrode, a wire, and multiple slugs in the second embodiment. [Figure 39] FIG. 39 is a diagram showing an example of input information, a processing example, and output information when the extraction targets are an electrode, a wire, and a plurality of slugs in the second embodiment. [Figure 40] FIG. 40 is a flowchart showing an example of the procedure of the learning process when the extraction targets are electrodes, wires, and slugs in the third embodiment. [Figure 41] FIG. 41 is a diagram showing an example of feature point data indicating actual feature points in the third embodiment. [Figure 42] FIG. 42 is a diagram illustrating an example of training data according to the third embodiment. [Figure 43] FIG. 43 is a diagram illustrating an example of the learning process in the third embodiment when the extraction target is slug. [Figure 44] FIG. 44 is a flowchart showing an example of the procedure of the inference process when the extraction target is an electrode, a wire, or a slug in the third embodiment. [Figure 45] FIG. 45 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode, a wire, or a slag in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0009] (First embodiment) 1 is a schematic diagram illustrating the configuration of a welding system including an information processing device according to the first embodiment. The welding system 1 includes an information processing device 10, a welding device 20, a storage device 30, a PoE (Power over Ethernet) hub 40, and a PLC (Programmable Logic Controller) 50.

[0010] The storage device 30 can be composed of any commonly used storage medium, such as a flash memory, a memory card, a RAM (Random Access Memory), an HDD (Hard Disc Drive), an SSD (Solid State Drive), and an optical disc.

[0011] Welding device 20 welds two or more members together. Welding device 20 performs, for example, arc welding or laser welding. Specific examples of arc welding include tungsten inert gas (TIG) welding, metal inert gas (MIG) welding, metal active gas (MAG) welding, and carbon dioxide arc welding. Here, we will mainly describe an example in which welding device 20 performs TIG welding.

[0012] Welding device 20 includes, for example, head 21, arm 22, power supply unit 26a, gas supply unit 26b, wire 23, imaging unit 24, lighting unit 25, and welding control device 26. Welding control device 26 includes power supply unit 26a, gas supply unit 26b, control unit 26c, and inference unit 26d. Welding control device 26 is an example of a control device.

[0013] A tungsten electrode 21a is provided on the head 21. The tip of the electrode 21a is exposed from the head 21. For example, the head 21 is attached to an articulated arm 22 including a plurality of links. Alternatively, the head 21 may be provided on a welding torch that is held by a worker.

[0014] Power supply unit 26a is electrically connected to electrode 21a and welding object S. Power supply unit 26a applies a voltage between electrode 21a and welding object S, causing an arc discharge. One of electrode 21a and welding object S may be set to a common potential (e.g., ground potential), and power supply unit 26a may control only the potential of the other of electrode 21a and welding object S.

[0015] The gas supply unit 26b is connected to the head 21. The gas supply unit 26b supplies an inert gas to the head 21. Alternatively, the gas supply unit 26b may supply a mixed gas of an inert gas and an active gas. The gas supplied to the head 21 is sprayed toward the welding object S from the tip of the head 21 where the electrode 21a is exposed.

[0016] The tip of the wire 23 is placed in the space where the arc discharge is occurring. The tip of the wire 23 melts due to the arc discharge and drips onto the welding object S. The molten wire 23 solidifies, thereby welding the welding object S. The wire 23 is fixed to the arm 22, for example, and is automatically fed as the wire 23 melts.

[0017] The imaging unit 24 captures an image of the welding area during welding. The imaging unit 24 captures an image of the welding area to obtain a still image. Alternatively, the imaging unit 24 may capture a video. The imaging unit 24 extracts a portion of the video to obtain a still image. The imaging unit 24 is, for example, a camera including a CCD image sensor or a CMOS image sensor.

[0018] The illumination unit 25 illuminates the welding area during welding so that a clearer image can be obtained by the imaging unit 24. If an image that can be used in subsequent processing can be obtained without illuminating the welding area, the illumination unit 25 does not need to be provided.

[0019] The inference unit 26d uses the feature point extraction model 1213 stored in the storage device 30 to execute an inference process for extracting feature points from the welding image. The feature point extraction model 1213 is a trained model trained by the information processing device 10, and the feature point extraction model 1213 trained by the information processing device 10 is copied to the storage device 30. Details of the feature point extraction model and the inference process will be described later.

[0020] Control unit 26c controls the operation of each of the above-mentioned components of welding device 20. For example, control unit 26c generates an arc discharge while driving arm 22, and welds welding object S along a predetermined direction.

[0021] Furthermore, the control unit 26c performs welding control based on information about the feature points extracted as a result of the inference by the inference unit 26d. The control unit 26c may also control the settings of the imaging unit 24, the lighting unit 25, and the like.

[0022] PoE hub 40 is a hub that connects imaging unit 24, information processing device 10, and PLC 50. PLC 50 is connected to welding control device 26 (or control unit 26c) of welding device 20.

[0023] The image captured by the imaging unit 24 is transmitted to the information processing device 10 via, for example, the PoE hub 40. The information processing device 10 receives the image captured by the imaging unit 24 and stores it in the storage device 30 connected to the information processing device 10. For example, the information processing device 10 associates the captured image with the welding conditions at the time of capturing and the capturing conditions at the time of capturing, and stores the associated images in the storage device 30.

[0024] The welding conditions include, for example, the applied voltage, gas flow rate, current value, wire feed speed, or welding speed. The photographing conditions include, for example, settings of the imaging unit 24, such as exposure time, aperture, or sensitivity (ISO). The photographing conditions may also include settings of the illumination unit 25. For example, when a pulse current is supplied to the illumination unit 25, the photographing conditions further include the pulse width, pulse frequency, duty ratio, or peak value. Note that, here, when multiple items are listed connected by "or," this means that all of the items may be included, or only some of the items may be included.

[0025] Fig. 2 is a block diagram showing an example of the configuration of the information processing device 10. As shown in Fig. 2, the information processing device 10 includes an acquisition unit 101, a teacher data generation unit 102, a learning unit 103, an output control unit 104, an input unit 105, a display unit 106, and a storage unit 121.

[0026] The storage unit 121 can be configured from any commonly used storage medium such as a flash memory, a memory card, a RAM, an HDD, an SSD, and an optical disc. The storage unit 121 stores various types of information used in various processes of the information processing device 10. For example, the storage unit 121 stores images acquired by the acquisition unit 101 and designation of correct answer data. In this embodiment, the storage unit 121 also stores an image information file 1211, training data 1212, and a feature point extraction model 1213. The information processing device 10 may be configured to use the storage device 30 instead of the storage unit 121.

[0027] The input unit 105 is an input device such as a keyboard, a mouse, a touch panel, etc. The display unit 106 is a display device such as a display device.

[0028] The acquisition unit 101 acquires various information used in the information processing device 10. For example, the acquisition unit 101 acquires one or more welding images (which may be either still images or videos) of an object to be welded, captured by the imaging unit 24 of the welding device 20. The acquisition unit 101 also acquires designation of correct answer data that represents true values ​​of the positions of multiple feature points in the welding image.

[0029] The ground truth data is used together with the welding image to machine-learn a feature point extraction model 1213 that detects the positions of specific feature points in an image, which is one of the image recognition technologies (image recognition tasks). That is, one or more pieces of training data 1212 including a welding image and true values ​​of the positions of multiple feature points in the welding image are used to train the feature point extraction model 1213.

[0030] Therefore, for example, the user specifies the positions of feature points in the image, the number of which is the number of feature points to be detected. The positions of the feature points may be specified by any method, for example, by directly inputting coordinate values ​​on the image, or by specifying the positions in the image with a mouse or the like using a separately provided graphical user interface. The acquisition unit 101 acquires ground truth data representing the positions of the feature points specified in this manner.

[0031] Acquisition unit 101 acquires welding images captured by welding device 20 from storage device 30. Acquisition unit 101 also extracts feature points from the acquired welding images and obtains position information (i.e., position coordinates) of the feature points. Acquisition unit 101 associates the storage location and file name of the welding image with the position information of the feature points of the welding image, and stores them in storage unit 121 as image information file 1211. The position information of the feature points of the welding image is called feature point information.

[0032] That is, the image information file 1211 is data in which the storage location (also referred to as address) of the image file of the welding image is associated with the feature point information of the welding image (i.e., position information and score) for each image file. The teacher data 1212 generated by the teacher data generation unit 102 includes the storage location and file name of the image file of the welding image and the true value of the position information, and is saved in the format of the image information file 1211. The storage location and file name of the image file of the welding image may be referred to as image file information.

[0033] The teacher data generation unit 102 generates one or more pieces of teacher data 1212 from the welding image and the positional information of the feature points obtained by the acquisition unit 101, or from the welding image obtained by the acquisition unit 101. Here, the teacher data 1212 is data having one or more welding images, feature point information including true values ​​(correct answer data) of the positional information of one or more feature points in the welding image, and event information. The event information is information indicating whether or not an event related to welding has occurred at a feature point (sometimes also referred to as "occurrence"). In this embodiment, the events specifically include welding-related events as feature points where the occurrence of the event is unspecified and the number of extractions is single, and welding-related events as feature points where the occurrence of the event is multiple, the number of occurrences is unspecified, and the number of extractions is multiple. The teacher data generation unit 102 stores the generated teacher data 1212 in the storage unit 121.

[0034] The learning unit 103 uses the training data 1212 generated by the training data generating unit 102 to train the feature point extraction model 1213 stored in the storage unit 121 .

[0035] The feature point extraction model 1213 is a model that detects the positions of specific feature points in an input image, which is one of the image recognition technologies. In this embodiment, the feature point extraction model 1213 inputs a welding image and outputs position information of the feature points and a score indicating the reliability of the position information. Here, the score is an example of an evaluation value.

[0036] Furthermore, the feature point extraction model 1213 also outputs event information indicating the presence or absence of an event in the case where the occurrence of an event is unspecified and the number of extracted feature points is single (an example of a first feature point), or in the case where the occurrence of an event is multiple, the number of occurrences is unspecified, and there are multiple extracted feature points (an example of a second feature point).

[0037] The feature point extraction model 1213 may be a model of any structure, for example, a model using DarkPose, which is one of the methods using a convolutional neural network. The learning unit 103 may be provided as a device separate from the information processing device 10.

[0038] In this embodiment, when no event occurs in the welding image, the learning unit 103 uses the teacher data 1212 to make the feature point extraction model 1213 learn that no event occurs. Furthermore, the learning unit 103 obtains a Gaussian distribution (normal distribution) from the positions of multiple feature points in the teacher data 1212, and uses the Gaussian distribution to train the feature point extraction model 1213.

[0039] In this embodiment, the characteristic points include characteristic points of any of the events of the electrode, the wire, the groove wall surface, and the contour of the molten pool, as well as characteristic points of any of the events of the droplet when the droplet falls, the hole when a digging defect occurs, and the hole when a burn-through occurs, which are examples of the first characteristic points.

[0040] Furthermore, when there are multiple occurrences of an event, the number of occurrences is unspecified, and there are multiple feature points to be extracted, the learning unit 103 obtains a predetermined distribution for each of the position information of the multiple feature points from the training data 1212, and uses a distribution obtained by combining the multiple predetermined distributions to learn the feature point extraction model 1213. Here, the combined distribution is an example of a second distribution.

[0041] A plurality of feature points where an event occurs multiple times but the number of occurrences is unspecified and the number of extracted features is multiple is an example of a second feature point, and includes any of slag indicating impurities generated during welding, abnormalities in the weld bead, and foreign matter adhering to the electrode.

[0042] Output control unit 104 controls the output of various information used in information processing device 10. For example, output control unit 104 outputs information (parameters, etc.) related to trained feature point extraction model 1213 to welding device 20. This allows welding device 20 to use trained feature point extraction model 1213 for controlling welding. For example, control unit 26c inputs an image (welding image) captured by imaging unit 24 to feature point extraction model 1213, and controls welding using the positions of multiple feature points output by feature point extraction model 1213. In addition, output control unit 104 performs display control on display unit 106.

[0043] Next, the overall process of learning the feature point extraction model 1213 by the information processing device 10 will be described. FIG. 3 is a flowchart showing an example of the overall learning process according to the first embodiment.

[0044] Acquiring unit 101 collects various types of information (S101). Specifically, acquiring unit 101 acquires one or more welding images captured by welding device 20 of welding system 1 from welding device 20. Then, acquiring unit 101 extracts feature points of each welding image, generates image information file 1211 in which image file information (storage location and file name of the image file) of the welding image is associated with the feature point information, and stores the image information file in storage unit 121.

[0045] Next, the teacher data generation unit 102 generates teacher data 1212 using the image information file 1211 stored in the storage unit 121 (S102). The teacher data 1212 is also stored in the storage unit 121.

[0046] Next, the learning unit 103 uses the training data 1212 to learn the feature point extraction model 1213 (S103), and the entire learning process ends.

[0047] The details of each of the above processes will be further explained below. First, a description will be given of examples of feature points used in welding system 1. Figures 4 to 6 are diagrams for explaining examples of feature points.

[0048] Fig. 4 shows an example of a welding image in which objects 501, 502, 503, and 504 corresponding to the above-mentioned four objects (OBJ1) to (OBJ4) are superimposed. Fig. 5 shows an example of a welding image in which feature points 601 and 602 corresponding to object 501 (electrode tip) and object 502 (wire tip) are detected. Fig. 6 shows an example of a welding image in which feature points 701a, 701b, 702a, and 702b corresponding to object 503 (groove wall surface) and feature points 711a, 711b, 712a, and 712b corresponding to object 504 (molten pool contour) are detected.

[0049] In this way, an object for which feature points are to be detected may be represented by one feature point or by multiple feature points. For example, as shown in Figure 5, the electrode tip and wire tip are each represented by one feature point. Also, as shown in Figure 6, the two groove walls (left and right groove walls) are each represented by two feature points, and the molten pool contour is represented by four feature points.

[0050] Note that strong light is emitted during welding, so two welding images, such as those shown in Figures 5 and 6, can be obtained by taking images using two modes with different exposure times. Alternatively, all feature points can be detected from a single welding image taken using a single exposure time. In this embodiment, a weld pool image including an image of a weld pool will be described as an example of a welding image, but an image other than a weld pool image may also be used as the welding image.

[0051] First, the information collection process performed by the welding system 1 and the acquisition unit 101 of the information processing device 10 in S101 of FIG. 3 will be described. FIG. 7 is a flowchart illustrating an example of a procedure of information collection processing according to the first embodiment.

[0052] The control unit 26c of the welding control device 26 turns on the imaging function and captures an image of the molten pool using the imaging unit 24 (S701). Next, the control unit 26c turns on the storage function and stores the molten pool image captured by the imaging unit 24 in the storage device 30 (S702).

[0053] Next, the acquisition unit 101 of the information processing device 10 turns on the feature point extraction function, acquires the molten pool image from the storage device 30, performs image processing on the molten pool image, and extracts the position information and scores of the feature points in the molten pool image (S703).

[0054] Specifically, the acquisition unit 101 extracts feature points from the molten pool image using an extraction model (not shown), which is a trained model prepared in advance. Furthermore, the acquisition unit 101 calculates a score, which is an index of the reliability of the position information of the feature points, i.e., an index of how confidently the extraction model has extracted the position information, based on the similarity between the inference data (i.e., the position information of the extracted feature points) and the teacher data 1212 that the feature point extraction model 1213 has learned so far. For example, if the inference data is similar to the teacher data 1212 that the feature point extraction model 1213 has learned so far, the score will be high, and if the feature point extraction model 1213 infers data that it has not learned, the score will be low.

[0055] The processes of S701 and S702 are repeatedly executed until the respective functions are stopped by the control unit 26c, and the process of S703 is repeatedly executed until the functions are stopped by the acquisition unit 101. The capture of the molten pool image in S701 and the storage of the molten pool image in S702 are executed at a processing speed of, for example, 20 fps. The processes of S701, S702, and S703 may be configured to be executed in parallel. In this case, the order of the processes of S702 and S703 does not matter.

[0056] Next, when the acquisition of the molten pool image and the extraction of the feature points are completed for a predetermined amount, the acquisition unit 101 stops the feature point extraction function (S704).The acquisition unit 101 then registers and saves the image file information (i.e., the storage location and file name) of the acquired molten pool image, which is the output result while the feature point extraction was functioning, and the extracted feature point position information in the image information file 1211 (S705).

[0057] Next, the control unit 26c stops the function of saving the molten pool image (S706). Next, the control unit 26c stops the function of capturing the molten pool image by the imaging unit 24 (S707). This ends the information collection process, and the process returns to the caller.

[0058] (Extraction target: electrode, wire, groove tip, molten pool contour) First, we will explain the case where the objects to be extracted from a welding image are the electrode, wire, groove tip, and weld pool contour. Only a single feature point of the electrode and wire is extracted as the extraction object, and there is no need to determine whether an event has occurred. Multiple feature points of the groove tip and weld pool contour are extracted as the extraction object, and there is no need to determine whether an event has occurred based on the score. Weld pool images containing feature points of the electrode, wire, groove tip, and weld pool contour are as shown in Figures 4 to 6.

[0059] Next, a detailed description will be given of the learning process of S103 for the feature point extraction model 1213. First, a description will be given of the learning process when the extraction targets are the electrode, wire, groove tip, molten pool contour, etc. shown in FIGS. FIG. 8 is a flowchart showing an example of the procedure of the learning process of the feature point extraction model 1213 when the extraction target is an electrode or a wire in the first embodiment.

[0060] First, the learning unit 103 inputs the training data generated from the image information file 1211 by the training data generating unit 102 and stored in the storage unit 121 (S801). FIG. 9 is a diagram showing an example of training data 1212 when the extraction target is an electrode or a wire in the first embodiment. As shown in Fig. 9, the teacher data 1212 is registered with an image and positional information of the electrode and wire in association with each other. Here, the image is a path indicating the location of the welding image file. The positional information is the position coordinates on the welding image.

[0061] FIG. 10 is a diagram for explaining an example of the learning process in the first embodiment when the extraction target is an electrode or a wire. 8, next, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrodes and wires for the image of the training data 1212 (S802), as shown in Fig. 10. Next, the learning unit 103 inputs the Gaussian distributions for the electrodes and wires to the feature point extraction model 1213, as shown in Fig. 10, and causes the feature point extraction model 1213 to learn (S803).

[0062] After the learning unit 103 has performed the above process on all images in the training data 1212, it copies the learned feature point extraction model 1213 to the storage device 30 and returns to the caller of the process.

[0063] Next, a welding control process using feature point extraction model 1213 by welding control device 26 will be described. FIG. 11 is a flowchart illustrating an example of a procedure of a welding control process according to the first embodiment.

[0064] The imaging unit 24 acquires images of the welding location, for example, in chronological order (S901). Next, the inference unit 26d executes an inference process for detecting a plurality of feature points from each acquired welding image using the trained feature point extraction model 1213 generated by the learning unit 103 of the information processing device 10 (S902). Then, the control unit 26c controls the welding by the welding device 20 so as to weld the optimal position based on the detected feature points as a result of the inference by the inference unit 26d (S903).

[0065] Any method for controlling welding based on feature points may be used. For example, a case will be described in which feature points corresponding to the following objects are used. (OBJ1) Tip of electrode 21a (hereinafter referred to as electrode tip) (OBJ2) Tip of wire 23 (hereafter referred to as wire tip) (OBJ3) Groove wall (OBJ4) Molten pool contour

[0066] In this case, the control unit 26c drives the arm 22 by feedback control so that the electrode tip and the wire tip are positioned at the center of the groove wall surface, for example. The control unit 26c also controls the speed at which the electrode tip and the wire tip are moved in a direction along the groove wall surface so that the contour of the molten pool comes into contact with the groove wall surface.

[0067] Next, the inference process of S902 will be described in detail. First, the inference process when the extraction target is an electrode or a wire will be described. FIG. 12 is a flowchart showing an example of the procedure of the inference process when the extraction target is an electrode or a wire in the first embodiment. FIG. 13 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode or a wire in the first embodiment.

[0068] 13, the inference unit 26d inputs each welding image to the feature point extraction model 1213. Then, the inference unit 26d uses the feature point extraction model 1213 to create a score distribution, which is a distribution of scores (an example of evaluation values) for each electrode and wire, for each welding image (S911).

[0069] Next, the inference unit 26d uses the feature point extraction model 1213 to determine the positions of each feature point of the electrodes and wires from the score distributions of the electrodes and wires generated in S911 (S912).

[0070] Then, inference unit 26d uses feature point extraction model 1213 to output the determined positions of each feature point of the electrode and wire and the scores at those positions as output information (S913). As shown in Fig. 13, output information including the position coordinates and scores of each of the electrode and wire in the welding image is output, and inference unit 26d passes the output information to control unit 26c. Then, the process returns to the caller.

[0071] In the above learning and inference processes, an example was given in which the extraction targets were electrodes and wires, but similar processes are also performed when the extraction targets are groove wall surfaces and molten pool contours.

[0072] (Extraction targets: droplets when droplets fall, holes when digging defects occur, holes when burn-through occurs) Next, we will explain the learning process (S103) and inference process (S902) when the feature points to be extracted from the molten pool image, which is a welding image, are droplets as they fall, holes as digging defects occur, and holes as burn-through occurs. When the feature points to be extracted are droplets as they fall, holes as digging defects occur, and holes as burn-through occur, the number of extracted feature points as feature points of the event that occurred is single, but whether or not they occur is unspecified, and for this reason, it becomes necessary to determine whether or not the extracted target exists using a score.

[0073] <Droplet falling from wire 23> The extraction target, "droplet falling from wire 23," will be explained. In the TIG welding of this embodiment, the insertion height of the wire 23 is subject to adjustment by the operator. Fig. 14 is a diagram showing an example of a molten pool image in the first embodiment when the insertion position of the wire 23 is appropriate, and the state of the insertion position of the wire 23. Fig. 14(a) shows the molten pool image, and Fig. 14(b) shows the state of the insertion position of the wire 23.

[0074] 15A and 15B are diagrams showing an example of the state of the insertion position of the wire 23 in the first embodiment. Fig. 15A shows a case where the insertion position of the wire 23 is normal, and Fig. 15B shows a case where the insertion position of the wire 23 is high.

[0075] 16A to 16D are diagrams showing examples of molten pool images when the insertion position of the wire 23 is high in the first embodiment. Each of Fig. 16A to 16D shows an example of a molten pool image when the insertion position of the wire 23 is high.

[0076] The area 1401 enclosed by a dotted line in the molten pool image in Figure 14(a) is the approximate range for the appropriate insertion position of the wire 23. If this wire insertion position is too low, the wire 23 will be separated from the arc 1402, resulting in insufficient heat input and the risk of the wire 23 remaining unmelted. Also, if the wire insertion position is too high, as shown in Figure 15(b), the wire 23 will not be inserted into the molten pool, and the arc 1402 will directly melt the wire 23, causing it to fall as droplets 1502.

[0077] The optimum height for the wire insertion position in the weld pool image cannot be uniquely determined; it varies depending on the installation position of the image capture unit 24, the welding conditions, and the welding posture. Since it is difficult to determine how far the wire insertion position should be lowered before residual molten metal is left, the wire 23 must be inserted at a position where droplets do not fall. Therefore, it becomes necessary to identify the droplets falling in the weld pool image. As shown in Figure 15(b), the state in which droplets 1502 hang from the wire tip can be recognized in the weld pool image as shown in Figure 16.

[0078] In this embodiment, the feature point extraction model 1213 is trained so that the droplet falling event in the molten pool image can be extracted as a feature point. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract the droplet falling event from the molten pool image as a feature point, and when the droplet falling event is extracted as a feature point, the insertion position of the wire 23 is determined to be high based on the position information, and welding control is performed to lower the insertion position of the wire 23.

[0079] <Digging defect> The "digging defect" that is the extraction target will be explained. 17A and 17B are diagrams showing an example of a state in which a digging defect occurs in the first embodiment. Fig. 17A shows a normal state in which no digging defect occurs, and Fig. 17B shows a state in which a digging defect occurs.

[0080] Fig. 18 shows an example of a molten pool image when a digging defect occurs in the first embodiment, Fig. 18(a) shows a normal case where no digging defect occurs, and Fig. 18(b) shows a case where a digging defect occurs.

[0081] In typical automatic TIG welding, as shown in FIG. 17(a), wire 23 is inserted into the molten pool while the base metal is melted by an arc. The arc pressure from the arc during TIG welding causes the molten pool directly below electrode 21a to become concave. If this arc pressure is high, the molten pool will become significantly depressed, as shown in the left diagram of FIG. 17(b). If the weld metal solidifies while covering this depressed area 1701, as shown in the right diagram of FIG. 17(b), this will result in an inherent defect in the weld metal. This type of defect is called a "dug-in defect."

[0082] In this way, the molten pool becomes a large concave shape and an digging defect is likely to occur, and the molten pool image will look like that shown in Figure 18(b).

[0083] Comparing this with the normal image of the molten pool shown in Figure 18(a), it is clear that a hole has opened up directly below the arc. Looking at this in a series of images (video), it is clear to an experienced worker that the hole is being dug up, and it is necessary to adjust the welding voltage or check for any problems with the wire feed.

[0084] For this reason, in this embodiment, the feature point extraction model 1213 is trained so that such engraving defect events in the weld pool image can be extracted as feature points. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract engraving defect events from the weld pool image as feature points, and once the engraving defect events are extracted as feature points, welding control such as adjusting the welding voltage is performed based on the position information.

[0085] <Melting off> The "burn-through" that is the subject of extraction will now be explained. Figure 19 shows an example of a state in which burn-through occurs and an example of a weld pool image in the first embodiment. Figure 19(a) shows an example of a groove shape. Figure 19(b) shows a normal state in which burn-through does not occur, and Figure 19(c) shows a state in which burn-through has occurred. Figure 19(d) shows an example of a weld pool image in a normal state in which burn-through does not occur, and Figures 19(e) to (g) show example weld pool images in a state in which burn-through has occurred.

[0086] Burn-through is a phenomenon similar to undercut defects. When narrowing the groove width for efficient welding in automatic TIG welding, an example groove shape like the one shown in Figure 19(a) is possible. When welding the first layer of components, if welding is carried out under appropriate welding conditions, the components can be joined while melting the first layer weld of the base material. On the other hand, if welding is carried out under inappropriate welding conditions (for example, high welding current or welding voltage, a slow welding wire feed speed, or a stop in welding wire feed), the first layer weld will melt and burn through without being joined. This burn-through is not limited to first layer welding, and care must be taken when the thickness of the welded area is thin (for example, the second or third layer).

[0087] If burn-through occurs, welding must be stopped immediately, the affected area repaired by manual welding, and automatic welding must be resumed. If welding is not stopped, the burn-through area may spread or the base material may come into contact with the electrode 21a, making it difficult to repair the affected area. Therefore, if holes caused by burn-through, such as digging defects, can be detected using image processing, welding can be stopped immediately and the effects of burn-through can be minimized.

[0088] As described above, in this embodiment, because the detection target is very similar to an undercut, a similar feature point extraction model 1213 is used to distinguish between an undercut defect and a burn-through defect based on the number of welding layers. That is, in this embodiment, the feature point extraction model 1213 is trained so that the burn-through phenomenon in the weld pool image can be extracted as a feature point. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract the burn-through phenomenon from the weld pool image as a feature point, and once the burn-through phenomenon has been extracted as a feature point, welding control such as stopping welding, repair, and resuming welding is performed based on the position information.

[0089] These phenomena—droplets during droplet drop, holes due to digging defects, holes due to burn-through, and slag, abnormal weld bead, and foreign matter adhering to the electrode 21a (discussed below)—directly affect welding quality and must be controlled or monitored by the welding system 1 as welding automation advances. Furthermore, these phenomena cannot be detected simply by extracting the positions of feature points using a trained feature point extraction model; instead, a function to distinguish between normal and abnormal welding images or to pinpoint abnormalities is required. For example, images of a weld pool without abnormalities, an image of the weld pool during droplet drop, and an image of the weld pool with floating slag are prepared. Each image is then assigned a flag indicating normal, droplet drop, or slag present. These images are then used as training data for learning, and a trained model for event discrimination is used to identify the condition.

[0090] When such an event determination model is used in the welding system 1, it may be possible to provide it separately from the feature point extraction model that outputs the positions of feature points, and to execute it in parallel with the feature point extraction model.

[0091] However, if the event determination model is provided separately from the feature point extraction model and executed in parallel in this way, it will affect the processing performance of the welding system 1, and specifications such as the timing of output information to be used for control when parallel processing will need to be considered. Furthermore, the feature point extraction model and the event determination model must be managed separately, and separate training data must also be prepared. Furthermore, when considering other applications or when re-training to address false detections, multiple types of training data are required, increasing the workload during operation.

[0092] For this reason, in this embodiment, as described above, the feature point extraction model 1213 is trained with training data so that it not only outputs the positions of feature points but also outputs the presence or absence of an event as event information, and this is used for inference.

[0093] Here, the training data includes the welding image, the true values ​​of the positions of the feature points, and the event information indicating whether or not an event is present.

[0094] For example, existing posture estimation models detect human joint points in input images. However, depending on the posture of the human, for example, arm joints may be hidden behind the torso and not appear in the image. In such cases, when training a posture estimation model, information about the hidden joint points is added to the training data rather than forcibly adding position information about the hidden joint points to the training data.

[0095] This approach to learning the posture estimation model is applied in adding training data of event information used for learning the feature point extraction model 1213 according to this embodiment. That is, in addition to cases where each of the following events occurs and appears in the welding image: droplets when droplets fall, holes when digging defects occur, holes when burn-through occurs, slag, abnormalities in the weld bead, and foreign matter adhering to the electrode 21a, which will be described later, there are also cases where these events do not occur and do not appear in the welding image.

[0096] Therefore, in this embodiment, when none of the above events occur, the teacher data generation unit 102 generates teacher data by marking the image as having no event, and the learning unit 103 uses this teacher data to train the feature point extraction model 1213.

[0097] As described above, the feature point extraction model 1213 outputs position information (position coordinates) and a score for each feature point. The score is an index of reliability, which indicates how confidently the feature point extraction model 1213 was able to output position information, and is also called confidence. For this reason, a feature point extraction model 1213 that has learned from a large amount of training data outputs position information for each feature point along with a high score, while a feature point extraction model 1213 that has learned from only a small amount of training data outputs position information for each feature point along with a low score. This score is useful for verifying proficiency in feature point extraction and identifying image patterns that are prone to false detection.

[0098] The feature point extraction model 1213 also outputs feature points of events that have been learned as no event, just as the posture estimation model described above outputs position information of hidden joint points.

[0099] That is, the feature point extraction model 1213 has a function for outputting position information of feature points such as the center of a droplet, the center of a slag, and the center of a dug hole, and can output each feature point with a high score for a welding image in which these events occur. On the other hand, when extracting feature points from a welding image in which these events do not occur, each feature point is output with a relatively low score because the information for each feature point is not included in the molten pool image. The control unit 26c of the welding control device 26 sets a threshold for the score and determines whether or not an event has occurred. That is, the control unit 26c determines that an event has occurred if the score is greater than the threshold, and that an event has not occurred if the score is less than the threshold.

[0100] Next, the learning process (S103) will be described when the extraction target is a droplet when the droplet falls, a hole when a carving defect occurs, or a hole when burn-through occurs. Note that, although the following description will use the phenomenon of a droplet when the droplet falls as an example, the same applies to the case of a hole when a carving defect occurs or a hole when burn-through occurs.

[0101] FIG. 20 is a flowchart showing an example of the procedure of the learning process in the first embodiment, in which the extraction targets are electrodes, wires, and droplets when droplets fall (hereinafter simply referred to as "droplets").

[0102] First, the learning unit 103 inputs the training data generated from the image information file 1211 by the training data generating unit 102 and stored in the storage unit 121 (S801). FIG. 21 is a diagram showing an example of training data 1212 in the first embodiment when the extraction targets are electrodes, wires, and falling droplets. As shown in Fig. 21, the teacher data 1212 is registered with an image and positional information of the electrode, wire, and droplets in association with each other. Here, the image is a path indicating the location of the welding image file. The positional information is the position coordinates on the welding image.

[0103] FIG. 22 is a diagram for explaining an example of the learning process in the first embodiment when the extraction targets are electrodes, wires, and droplets. Returning to Fig. 20, next, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrode, wire, and droplets for images in the teacher data 1212 that contain droplets (i.e., images in which a droplet event occurs) as shown in Fig. 22 (S1001). Next, the learning unit 103 inputs the Gaussian distributions for the electrode, wire, and droplets into the feature point extraction model 1213 as shown in Fig. 22(a), and causes the feature point extraction model 1213 to learn (S1002).

[0104] In parallel with the above processing, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrode and wire, respectively, for images without droplets (i.e., no droplet events) in the teacher data 1212, as shown in Fig. 22 (S1003). Next, the learning unit 103 inputs the Gaussian distributions of the electrode and wire into a feature point extraction model 1213, as shown in Fig. 22(b), and causes the feature point extraction model 1213 to learn (S1004).

[0105] Next, the learning unit 103 inputs to the feature point extraction model 1213 that there are no droplets in the target image (that is, the image without droplets), and causes the feature point extraction model 1213 to learn (S1005).

[0106] After the learning unit 103 has performed the above process on all images in the training data 1212, it copies the learned feature point extraction model 1213 to the storage device 30 and returns to the caller of the process.

[0107] Next, the inference process (S902) will be described when the extraction target is a droplet when the droplet falls, a hole when a digging defect occurs, or a hole when burn-through occurs. Note that, although the following description will use the phenomenon of a droplet when the droplet falls as an example, the same applies to the case of a hole when a digging defect occurs or a hole when burn-through occurs.

[0108] FIG. 23 is a flowchart showing an example of the procedure of the inference process when the extraction target is an electrode, a wire, or a droplet in the first embodiment. FIG. 24 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode, a wire, or a droplet in the first embodiment.

[0109] 24, the inference unit 26d inputs each welding image to the feature point extraction model 1213. Then, the inference unit 26d uses the feature point extraction model 1213 to create a score distribution of each score for each of the electrode, wire, and droplet for each welding image (S1101).

[0110] Next, the inference unit 26d uses the feature point extraction model 1213 to determine the positions of the feature points of the electrode, wire, and droplets from the score distributions of the electrode, wire, and droplets generated in S1101 (S1102).

[0111] Next, the inference unit 26d uses the feature point extraction model 1213 to determine the presence or absence of droplets from the distribution of scores of the droplet positions (S1103). Next, the inference unit 26d outputs, as output information, position information of each of the determined feature points of the electrode, wire, and droplets, scores at the positions, and the presence or absence of droplets (S1104) using the feature point extraction model 1213. As shown in Fig. 24, output information including the position coordinates, scores, and presence or absence of droplets of each of the electrode, wire, and droplets in the welding image is output, and the inference unit 26d passes the output information to the control unit 26c.

[0112] (Extraction target: slag, abnormal welding beads, foreign matter adhering to the electrode 21a) Next, we will explain the learning process (S103) and inference process (S902) when the feature points to be extracted from the molten pool image are slag, anomalies in the weld bead, or foreign matter adhering to the electrode. When the feature points to be extracted are slag, anomalies in the weld bead, or foreign matter adhering to the electrode, there are multiple occurrences of the event, the number of occurrences is unspecified, and there are multiple extractions. For this reason, it becomes necessary to determine whether or not the extraction target exists using a score.

[0113] <Slag> The extraction target, "slag," will now be explained. Slag is an impurity that is generated during welding. Slag floats on the molten pool during welding and can be recognized in molten pool images.

[0114] Figure 25 shows an example of an image of a molten pool with slag in the first embodiment. In Figure 25, slag is observed at the locations indicated by the arrows. Figures 25(a) and (b) show an example with one slag, while Figures 25(c) and (d) show an example with two slags.

[0115] It is difficult to control the generation of slag, and there is no procedure or adjustment work such as stopping welding when slag occurs. For this reason, it is not subject to control by the welding system 1. On the other hand, information that slag has occurred is useful, and it is expected that the presence or absence of slag will be recorded in the welding system 1 log, an alarm will be issued, or other operations will be performed.

[0116] If the slag floating on the surface of the molten pool disappears, i.e., if it can no longer be recognized in the molten pool image, it is possible that it has adhered to the surface of the solidified metal or that it is contained within the solidified metal. Therefore, if the slag on the molten pool disappears and cannot be recognized on the appearance of the bead after welding, there is a concern that the slag may be contained within the solidified metal.

[0117] Fig. 26 is a diagram showing an example of repair work when slag is identified in the first embodiment. Fig. 26(a) shows an example of repair work in a normal case, and (b) shows an example of repair work in a case where slag can be identified by image processing.

[0118] Normally, when the presence of slag is identified through non-destructive testing after the completion of the welding process, repair work involves removing all of the weld metal up to the location (i.e., depth) of the slag, as shown in Figure 26(a). However, if the presence or absence of slag can be confirmed in a molten pool image and the possibility of its presence is suggested, it can be dealt with by simple repair work after welding, as shown in Figure 26(b).

[0119] In this embodiment, the feature point extraction model 1213 is trained so that the phenomenon of slag generation in the molten pool image can be extracted as a feature point. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract the slag phenomenon from the molten pool image as a feature point, and once the slag phenomenon is extracted as a feature point, repair work is performed based on the position information.

[0120] <Adhesion of foreign matter to electrode 21a> The extraction target, "foreign matter adhering to the electrode 21a," will be described. Figure 27 shows examples of molten pool images in the first embodiment, showing the normal welding state, the state of foreign matter adhering to the electrode 21a, and when foreign matter has adhered to the electrode 21a. Figure 27(a) shows the normal welding state, and Figure 27(b) shows the state of foreign matter adhering to the electrode 21a. Figures 27(c) to (f) show examples of molten pool images when foreign matter has adhered to the electrode 21a.

[0121] In automatic TIG welding, as shown in FIG. 27(a), an arc is generated from the electrode 21a toward the base material, melting the base material and welding wire while welding. If welding is continued for a long period of time, foreign matter adheres and accumulates around the tip of the electrode 21a, as shown in FIG. 27(b). The amount of foreign matter that adheres and the rate at which it accumulates vary depending on the type of electrode 21a, the base material, the welding wire materials, and the welding conditions. As foreign matter adheres and accumulates, the spread of the arc is inhibited, raising concerns about reduced penetration. For this reason, when a certain amount of foreign matter adheres around the tip of the electrode 21a, the electrode 21a is replaced or polished at the timing of stopping welding, and welding is resumed using an electrode 21a that is free of foreign matter.

[0122] When welding rotating cylindrical components such as pipes or rotors, it is possible to continue welding for long periods of time as long as there are no problems with the base material or welding materials. When applying automatic control to welding system 1 for long-term continuous welding, factors that force welding to be stopped and adjustments to the setup include wear of electrode 21a and adhesion of foreign matter. If foreign matter adhesion to electrode 21a can be detected using molten pool images and image processing, as shown in Figures 27(c) to (f), it is possible to automatically notify the system when it is time to replace the electrode. Furthermore, if foreign matter adhesion to electrode 21a significantly affects the spread of the arc, it can cause insufficient fusion, so welding defects can be prevented by automatically stopping the welding.

[0123] In this embodiment, the feature point extraction model 1213 is trained so that the phenomenon of foreign matter adhering to the electrode 21a in the molten pool image can be extracted as a feature point. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract the phenomenon of foreign matter adhering to the electrode 21a from the molten pool image as a feature point, and once the phenomenon of foreign matter adhering to the electrode 21a is extracted as a feature point, welding control such as issuing an announcement or stopping the welding process is performed based on the position information.

[0124] <Weld bead abnormalities> The extraction target, "abnormality in weld beads," will now be described. When capturing an image of the molten pool, setting the exposure time long to capture a bright image allows you to check the appearance of the weld bead.

[0125] FIG. 28 shows an example of an image of a weld pool or a cross section of a weld bead related to an abnormality in the weld bead in the first embodiment. FIG. 28(a) shows an example of a normal welding state, and FIG. 28(b) shows an example of a normal weld bead cross section. FIG. 28(c) shows an example of a state of a poor weld bead shape and poor fusion, and FIG. 28(d) shows an example of a weld bead cross section in the case of a poor weld bead shape and poor fusion. FIG. 28(e) shows an example of a pit state, and FIG. 28(f) shows an example of a weld bead cross section in the case of a pit.

[0126] In the normal weld bead shown in Figure 28(a), there is no abnormality on the bead surface, and as shown in Figure 28(b), the boundary between the groove wall surface and the bead surface is smoothly welded in a rounded shape.

[0127] On the other hand, abnormal conditions can also be confirmed by the appearance of the weld bead. For example, there is lack of fusion caused by poor weld bead shape, as shown in Figure 28(c). In this case, as shown in the weld bead cross section in Figure 28(c), if the boundary between the groove wall and the weld bead is not smooth and rounded, but is welded at an acute angle, there is a risk that the acute angle will not be fully melted when welding the next layer. This remaining unmelted area is a welding defect known as lack of fusion, and poor weld bead shape is cited as one of the causes of lack of fusion.

[0128] Such areas with poor weld bead shape can be confirmed on the molten pool image. As shown by the arrow in the right-hand image of Figure 28(d), differences in the reflection of the arc light occur at areas with poor weld bead shape between the groove wall and the weld bead boundary, resulting in different brightness at the relevant areas. If areas with poor weld bead shape can be extracted using image processing, welding defects can be prevented by stopping welding before areas that could result in poor fusion can be welded. Furthermore, if such areas can be extracted from the weld bead after welding, the presence or absence of abnormalities can be checked in the log before the next layer is welded, and the bead shape can be formed before welding the next layer begins.

[0129] Pits, as shown in Figure 28(e), are welding defects that can be identified from poor weld bead shape. Bubbles form in the molten pool due to poor shielding gas or impurities in the weld area. There is no problem if the bubbles are expelled from the molten pool before the weld metal solidifies, but if the bubbles solidify before they are expelled, voids remain in the weld metal. Weld defects like this are called blowholes, and depending on the size and number of affected areas, they may require welding repair. However, there are also cases where the weld metal solidifies while the bubbles are being expelled from the molten pool. As shown in Figure 28(f), this is a welding defect in the form of bubbles appearing on the bead surface, and is called a pit.

[0130] There is a possibility that blowholes may exist around the pit, and considering the need to ensure the integrity of the next layer's weld and the risk of repairing the defect later, the relevant area should be removed before welding. If the relevant area can be extracted using image processing, welding can be stopped beforehand to prevent welding defects. Furthermore, if pits can be extracted from the weld bead after welding, the presence or absence of pits can be checked in the log before welding the next layer, and the relevant area can be removed before welding the next layer can begin.

[0131] In this embodiment, the feature point extraction model 1213 is trained so that such an event of a weld bead abnormality in a molten pool image can be extracted as a feature point. Then, in this embodiment, the trained feature point extraction model 1213 is used to extract the event of a weld bead abnormality from the molten pool image as a feature point, and once the event of a weld bead abnormality is extracted as a feature point, welding control is performed based on the position information.

[0132] If the event is a single event, such as droplet fall, the control unit 26c of the welding control device 26 can determine whether or not the event exists by judging the score using a threshold. However, events such as slag, abnormalities in the weld bead, and foreign matter adhering to the electrode can be extracted in multiple, unspecified numbers. For this reason, a simple feature point extraction process will output multiple peak positions in the score distribution. Feature point extraction is performed as follows for the detection of events such as slag, and for events that can be observed multiple times on a molten pool image.

[0133] To manage slag-related feature points, a large number of slag feature points are prepared in advance, and only those with slag are assigned position information indicating an event (i.e., ON), while the remaining slag feature points are assigned position information indicating no event (i.e., OFF). For example, as shown in Figure 31 (described later), five slag feature points (SL1-5) are used. In a molten pool image without slag, SL1-5 are assigned position information indicating no event (OFF). In a molten pool image with slag, as shown in Figure 31, position information is assigned to only the relevant number of feature points. In the example of Figure 31(b), since there is one slag, only SL1 is assigned the slag center position, and SL2-5 are assigned position information indicating no event (OFF). In Figure 31(c), since there are two slags, SL1 and SL2 are assigned position information indicating the slag center position, and SL3-5 are assigned position information indicating no event (OFF). The training data generation unit 102 creates training data with this configuration of training points.

[0134] In the feature point extraction model 1213, which was trained on these training data in the usual way, for a molten pool image with one slag, as shown in Figure 31(b), the feature points SL1 to SL5 give high scores around the slag, although there are differences. Also, for a molten pool image with multiple slags, as shown in Figure 31(d), the feature points SL1 to SL5 give high scores around each slag. Even if each of these output information is processed separately, it is difficult to distinguish information such as how many slags there are and which slags SL1 to SL5 each are.

[0135] Therefore, when the number of extraction targets is unspecified, the learning unit 103 according to this embodiment integrates each of the Gaussian distributions SL1 to SL5 into one distribution, and the feature point extraction model 1213 learns one distribution for the entire slag. Additionally, the learning unit 103 is configured so that the feature point extraction model 1213 outputs one distribution for the slag. The control unit 26c of the welding control device 26 sets a threshold for the distribution of scores output from the inference unit 26d, and determines positions equal to or greater than the threshold as slag.

[0136] Next, the learning process (S103) will be described when the extraction target is slag, an abnormality in the weld bead, or a foreign object adhering to electrode 21a. Note that, although the following description will be given using multiple slag events as examples, the same applies to the cases of an abnormality in the weld bead and a foreign object adhering to electrode 21a. FIG. 29 is a flowchart showing an example of the procedure of the learning process in the first embodiment when the extraction targets are an electrode, a wire, and multiple slugs.

[0137] First, the learning unit 103 inputs the training data generated from the image information file 1211 by the training data generating unit 102 and stored in the storage unit 121 (S801).

[0138] FIG. 30 is a diagram showing an example of training data 1212 in the first embodiment when the extraction target is an electrode, a wire, or a slug. As shown in Fig. 30, the teacher data 1212 is registered with an image and positional information of the electrode, wire, and slag in association with each other. Here, the image is a path indicating the location of the welding image file. The positional information is the position coordinates on the welding image.

[0139] In addition, there are multiple slags, and the number of slags varies from image to image. In the example of the welding image in this training data, the maximum number of slags is five, and for welding images with less than five slags, information indicating no slag is registered in the training data. 31 is a diagram showing an example of a welding image in which a plurality of slags exist in the first embodiment, where SLn (n is an integer) indicates the slags.

[0140] In the welding image of Figure 31(a), there is one slag. As shown in the same welding image of Figure 31(b), the slag is indicated by SL1. Therefore, the teacher data generation unit 102 registers the position information of SL1 as event ON in the teacher data 1212. On the other hand, the teacher data generation unit 102 registers no slag (event OFF) in the teacher data 1212 for SL2 to SL5.

[0141] In the welding image of Figure 31(c), there are two slags. As shown in the same welding image of Figure 31(d), the slags are indicated by SL1 and SL2. For this reason, the teacher data generation unit 102 registers the event ON for SL1 and SL2 in the teacher data 1212, and registers their respective position information. On the other hand, the teacher data generation unit 102 registers no slag (event OFF) for SL3 to SL5 in the teacher data 1212.

[0142] FIG. 32 is a diagram for explaining an example of the learning process in the first embodiment when the extraction targets are electrodes, wires, and slags. Returning to FIG. 29, next, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrode, the wire, and the plurality of slugs in the teacher data 1212, as shown in FIG. 32 (S1201).

[0143] Next, the learning unit 103 generates a slag distribution by combining the Gaussian distributions of the multiple slags (S1202), as shown in Fig. 32. Next, the learning unit 103 inputs the Gaussian distributions of the electrode, wire, and slag (the slag distribution is the combined distribution) into the feature point extraction model 1213, as shown in Fig. 32, and causes the feature point extraction model 1213 to learn (S1203).

[0144] Next, if there are no slugs in the image (including when the maximum number is not reached), the learning unit 103 inputs the fact that there are no slugs into the feature point extraction model 1213 and trains the feature point extraction model 1213 (S1205).

[0145] After the learning unit 103 has performed the above process on all images in the training data 1212, it copies the learned feature point extraction model 1213 to the storage device 30 and returns to the caller of the process.

[0146] Next, the inference process (S902) will be described when the extraction target is slag, an abnormality in the weld bead, or a foreign object adhering to the electrode 21a. Note that, although the following description uses multiple slag events as examples, the same applies to the cases of an abnormality in the weld bead and a foreign object adhering to the electrode 21a.

[0147] FIG. 33 is a flowchart showing an example of the procedure of the inference process when the extraction targets are an electrode, a wire, and multiple slugs in the first embodiment. FIG. 34 is a diagram showing an example of input information, a processing example, and output information when the extraction targets are an electrode, a wire, and a plurality of slugs in the first embodiment.

[0148] 34, the inference unit 26d inputs each welding image to the feature point extraction model 1213. As a result, the inference unit 26d uses the feature point extraction model 1213 to create a score distribution of each score for each electrode, wire, and droplet for each welding image (S1301).

[0149] Next, the inference unit 26d determines whether or not there is a slug based on the distribution of slug scores using the feature point extraction model 1213 (S1302). Next, the inference unit 26d determines the positions of the electrodes, wires, and each feature point from the score distributions of the electrodes and wires generated in S1301, using the feature point extraction model 1213 (S1303).

[0150] Next, if there is a slug, the inference unit 26d determines the position of the feature point of the slug from the score distribution of the slug using the feature point extraction model 1213 (S1305). Next, the inference unit 26d uses the feature point extraction model 1213 to output, as output information, position information of each feature point of the determined electrodes and wires, the score at that position, and the presence or absence of a slug (S1306). Next, using the feature point extraction model 1213, if there are slugs, the inference unit 26d adds the number of slugs, position information, and the score at that position to the output information and outputs it (S1308). As shown in FIG. 34, output information including the position coordinates and scores of the electrode, wire, and slag in the welding image is output, and the inference unit 26d passes the output information to the control unit 26c.

[0151] As described above, the information processing device 10 according to this embodiment includes a learning unit 103 that uses training data 1212 including one or more welding images of an object to be welded, true values ​​of position information of one or more feature points in the welding image, and event information indicating whether or not a welding-related event has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of which is single, to learn a feature point extraction model 1213 that inputs a welding image and outputs the position information of the feature points, a score indicating the reliability of the position information, and the event information.

[0152] For this reason, according to this embodiment, in addition to learning the position information of the feature points, the feature point extraction model 1213 is trained using training data 1212 including event information indicating whether or not a welding-related event has occurred as a first feature point, which is one or more feature points and for which the occurrence of the event is unspecified and which is extracted in a single number. More specifically, when the first feature point does not exist in the welding image, the training unit 103 uses the training data 1212 to train the feature point extraction model 1213 that no event has occurred. Therefore, according to this embodiment, the feature point extraction model can be properly trained for events of various feature points, thereby enabling appropriate control of welding.

[0153] In the information processing device 10 according to this embodiment, the learning unit 103 obtains a Gaussian distribution for a plurality of feature points from the training data 1212, and uses the Gaussian distribution to learn the feature point extraction model 1213. Therefore, according to this embodiment, by learning the Gaussian distribution, the feature point extraction model can be properly trained for various feature point phenomena, thereby enabling appropriate control of welding.

[0154] In the information processing device 10 according to this embodiment, the learning unit 103 obtains a Gaussian distribution for each of a plurality of second feature points, among the plurality of feature points, where multiple events occur but the number of occurrences is unspecified and where the number of extractions is multiple, from the teacher data 1212, and learns the feature point extraction model using a distribution (second distribution) obtained by combining the multiple Gaussian distributions. Therefore, in this embodiment, by learning the distribution obtained by combining Gaussian distributions for each of the plurality of feature points (second feature points) where multiple events occur but the number of occurrences is unspecified and where the number of extractions is multiple, the feature point extraction model can be properly trained for events of various feature points, thereby enabling appropriate control of welding.

[0155] The welding control device 26 according to this embodiment is equipped with an inference unit 26d that uses a feature point extraction model 1213 to input one or more welding images of an object to be welded and output position information of one or more feature points in the welding image, a score indicating the reliability of the position information, and event information indicating whether or not an event related to welding has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of extractions of which is single, and outputs the position information, score, and event information of the feature points from the welding image; and a control unit 26c that controls welding based on the position information, score, and event information of the feature points output by the inference unit 26d.

[0156] For this reason, according to this embodiment, welding is controlled based on the position information of the feature points, the scores, and the event information output using a feature point extraction model 1213 that outputs not only the position information of the feature points but also event information indicating whether or not an event related to welding has occurred as a feature point (first feature point) for which the occurrence of the event is unspecified and the number of extractions is single among multiple feature points. Therefore, according to this embodiment, by performing inference using such a feature point extraction model, various feature points can be extracted and welding can be appropriately controlled.

[0157] Furthermore, in welding control device 26 according to this embodiment, inference unit 26d outputs the presence or absence of an event occurring based on the distribution of feature point scores using feature point extraction model 1213. Therefore, according to this embodiment, the presence or absence of an event occurring is inferred based on the distribution of feature point scores, and therefore various feature points can be extracted to more appropriately control welding.

[0158] Furthermore, in welding control device 26 according to this embodiment, inference unit 26d outputs the presence or absence of an event occurring based on the distribution of scores of a plurality of feature points (second feature points) among the plurality of feature points, where the number of occurrences of an event is unspecified and the number of extractions is multiple, using feature point extraction model 1213. Therefore, according to this embodiment, the presence or absence of an event occurring is inferred based on the distribution of feature point scores, and thus various feature points can be extracted to more appropriately control welding.

[0159] In the information processing device 10 and welding control device 26 according to this embodiment, among one or more feature points, a feature point (first feature point) for which the occurrence of an event is unspecified and for which the number of extractions is single includes any of a droplet when the droplet falls, a hole when a digging defect occurs, and a hole when burn-through occurs. Therefore, according to this embodiment, the feature point extraction model can be properly trained for various feature point events, such as a droplet when the droplet falls, a hole when a digging defect occurs, and a hole when burn-through occurs. Therefore, by performing inference using the feature point extraction model trained in this way, various feature points, such as a droplet when the droplet falls, a hole when a digging defect occurs, and a hole when burn-through occurs, can be extracted, and welding can be properly controlled.

[0160] In the information processing device 10 and welding control device 26 according to this embodiment, among the plurality of feature points, the plurality of feature points (second feature points) in which the number of occurrences of events is unspecified and the number of extracted feature points is multiple include slag indicating impurities generated during welding, anomalies in the weld bead, and foreign matter adhering to the electrode. Therefore, according to this embodiment, the feature point extraction model can be appropriately trained for various feature point events such as slag indicating impurities generated during welding, anomalies in the weld bead, and foreign matter adhering to the electrode. Therefore, by performing inference using the trained feature point extraction model, various feature points such as slag indicating impurities generated during welding, anomalies in the weld bead, and foreign matter adhering to the electrode can be extracted, thereby enabling appropriate control of welding.

[0161] (Second embodiment) In the first embodiment, when a welding image (i.e., a molten pool image) contains a plurality of feature points (second feature points) for which an event occurs multiple times, the number of occurrences is unspecified, and the number of extractions is multiple, such as feature points for events such as foreign matter adhering to the electrode 21a and multiple unspecified slag, the position information and the presence or absence of the event are input to the feature point extraction model 1213, and learning is performed, and inference is performed using the trained feature point extraction model 1213. In the second embodiment, when a welding image contains a plurality of feature points for which an event occurs multiple times, the number of occurrences is unspecified, and the number of extractions is multiple, the position of the feature points, the presence or absence of the event, and the size of the feature points are input to the feature point extraction model 1213, and learning is performed, and inference is performed using the trained feature point extraction model 1213.

[0162] The configurations of the welding system 1, information processing device 10, and welding control device 26 according to this embodiment are the same as those of the first embodiment. In the information processing device 10 of this embodiment, the learning unit 103 has the same functions as in the first embodiment, and in addition, when a welding image contains multiple feature points (second feature points) where multiple events occur but the number of occurrences is unspecified and the number of extractions is multiple, the learning unit 103 further learns a feature point extraction model 1213 using training data 1212 including the sizes of the multiple feature points (second feature points).

[0163] Furthermore, in the welding control device 26 according to this embodiment, the inference unit 26d has the same functions as those in the first embodiment, and in addition, when the welding image contains a plurality of feature points (second feature points) in which an event has occurred but the number of occurrences is unspecified and the number of extractions is multiple, the inference unit 26d uses the feature point extraction model 1213 to output the size of the feature points when an event has occurred, based on the distribution of the scores of the plurality of feature points (second feature points).

[0164] Here, in the welding image, when multiple events occur but the number of occurrences is unspecified and multiple extracted feature points (second feature points) include either slag or foreign matter adhering to the electrode. Note that when the feature points are foreign matter adhering to the electrode, the size may be expressed as an approximate size such as large, medium, or small.

[0165] Next, the learning process (S103) will be described when the extraction target is slag or foreign matter adhering to the electrode 21a. Note that, although the following description will be given using an example of multiple slag events, the same applies to foreign matter adhering to the electrode 21a. FIG. 35 is a flowchart showing an example of the procedure of the learning process in the second embodiment when the extraction targets are an electrode, a wire, and a plurality of slugs.

[0166] First, the learning unit 103 inputs the training data generated from the image information file 1211 by the training data generating unit 102 and stored in the storage unit 121 (S801).

[0167] FIG. 36 is a diagram showing an example of training data 1212 in the case where the extraction target is an electrode, a wire, or a slug in the second embodiment. As shown in Fig. 36, the teacher data 1212 is registered with an image and positional information of the electrode, wire, and slag in association with each other. Here, the image is a path indicating the location of the welding image file. The positional information is the position coordinates on the welding image.

[0168] As in the first embodiment, there are multiple slugs, and the number of slugs varies for each image and is unspecified. In addition to the position information, the size of the slug is also registered.

[0169] FIG. 37 is a diagram for explaining an example of the learning process in the second embodiment when the extraction targets are electrodes, wires, and slags. Returning to FIG. 35, next, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrode, the wire, and the plurality of slugs in the teacher data 1212, as shown in FIG. 37 (S1401).

[0170] Next, the learning unit 103 generates a Gaussian distribution based on the positions and sizes of the plurality of slugs (S1402). Next, the learning unit 103 generates a slug distribution (second distribution) by combining the Gaussian distributions of the plurality of slugs (S1403).

[0171] Next, as shown in FIG. 37, the learning unit 103 inputs the Gaussian distributions of the electrode and the wire slug (the distribution of the slug is a combined distribution) into the feature point extraction model 1213, and causes the feature point extraction model 1213 to learn (S1404).

[0172] Next, if there are no slugs in the image (including when the maximum number is not reached), the learning unit 103 inputs the fact that there are no slugs into the feature point extraction model 1213 and trains the feature point extraction model 1213 (S1406).

[0173] After the learning unit 103 has performed the above process on all images in the training data 1212, it copies the learned feature point extraction model 1213 to the storage device 30 and returns to the caller of the process.

[0174] Next, the inference process (S902) will be described when the extraction target is slag or foreign matter adhering to the electrode 21a. Note that, although the following description will be given using an example of multiple slag events, the same applies to foreign matter adhering to the electrode 21a.

[0175] FIG. 38 is a flowchart showing an example of the procedure of the inference process when the extraction targets are an electrode, a wire, and multiple slugs in the second embodiment. FIG. 39 is a diagram showing an example of input information, a processing example, and output information when the extraction targets are an electrode, a wire, and a plurality of slugs in the second embodiment.

[0176] 39, the inference unit 26d inputs each welding image to the feature point extraction model 1213. The inference unit 26d uses the feature point extraction model 1213 to create a score distribution of each score for each of the electrode, wire, and droplet (S1501), determine the presence or absence of slag (S1502), and determine the positions of the electrode, wire, and each feature point (S1503), which are performed in the same manner as S1301, S1302, and S1303 in the first embodiment.

[0177] Next, if there is a slug, the inference unit 26d uses the feature point extraction model 1213 to determine the position and size of the feature point of the slug from the score distribution of the slug (S1505). Next, the inference unit 26d uses the feature point extraction model 1213 to output, as output information, position information of each feature point of the determined electrodes and wires, the score at that position, and the presence or absence of a slug (S1506).

[0178] Next, using the feature point extraction model 1213, if there are slugs, the inference unit 26d adds the number of slugs, position information, score at that position, and size to the output information and outputs it (S1508).

[0179] As shown in FIG. 39, output information including the position coordinates and scores of the electrodes and wires in the welding image, and the position information, scores, and sizes of each of the multiple slags is output, and the inference unit 26d passes the output information to the control unit 26c.

[0180] As described above, in the information processing device 10 according to this embodiment, in addition to the functions of the first embodiment, the learning unit 103 learns a feature point extraction model using training data including the sizes of a plurality of second feature points, among the plurality of feature points, where multiple events occur but the number of occurrences is unspecified and a plurality of feature points are extracted. Therefore, in this embodiment, the feature point extraction model can be properly trained for various feature point events, including the feature point sizes, and various feature points can be extracted together with their sizes to more appropriately control welding.

[0181] Furthermore, in welding control device 26 according to this embodiment, in addition to the functions of the first embodiment, inference unit 26d uses a feature point extraction model to output the size of the feature point (second feature point) when an event has occurred, based on the distribution of scores. Therefore, according to this embodiment, inference is performed using feature point extraction model 1213 that has been trained including size, thereby extracting various feature points and more appropriately controlling welding.

[0182] In the information processing device 10 and welding control device 26 according to this embodiment, among the plurality of feature points, the plurality of feature points (second feature points) in which the number of occurrences of multiple events is unspecified and the number of extracted feature points is multiple include either slag indicating impurities generated during welding or foreign matter adhering to the electrode. Therefore, according to this embodiment, it is possible to appropriately train the feature point extraction model on various feature point events, such as slag indicating impurities generated during welding and foreign matter adhering to the electrode, along with their sizes. Therefore, by performing inference using the feature point extraction model trained in consideration of size, various feature points, such as slag indicating impurities generated during welding and foreign matter adhering to the electrode, can be extracted, and welding can be appropriately controlled.

[0183] (Third embodiment) In the first embodiment, the learning process and the inference process are described for a case where the occurrence of an event is unspecified and a single extracted first feature point is present in a welding image. Also, in the first embodiment and the second embodiment, the learning process and the inference process are described for a case where a plurality of events occur, the number of occurrences is unspecified, and a plurality of extracted second feature points are present.

[0184] In this third embodiment, a learning process and an inference process will be described for a case where a welding image contains a feature point in which multiple events occur but the number of occurrences is unspecified and a single feature point is extracted.

[0185] The configurations of the welding system 1, information processing device 10, and welding control device 26 according to this embodiment are the same as those of the first embodiment. In the information processing device 10 of this embodiment, the learning unit 103 has the same functions as in the first embodiment, and also learns a feature point extraction model 1213 using training data 1212 that includes, among one or more feature points in a welding image, a feature point (an example of a third feature point) where an event occurs multiple times but the number of occurrences is unspecified and the number of extractions is single.

[0186] In this embodiment, although the number of occurrences of events is unspecified, the number of extractions is single, so the learning unit 103 only needs to determine the distribution of the slag and use that distribution to learn the feature point extraction model 1213. There is no need to further determine the distribution of the entire slag by combining the distributions of multiple slags, as in the slag example (second feature point) in the first embodiment. Therefore, the specific learning method is the same as the learning process in the first embodiment when the occurrence of an event such as a droplet is unspecified and there is a first feature point with a single extraction number.

[0187] Furthermore, when a welding image contains a plurality of predetermined events, each occurring multiple times and with an unspecified number of occurrences, the teacher data generation unit 102 according to this embodiment extracts a single event from the plurality of predetermined events in the welding image as a feature point (third feature point) based on a predetermined criterion, and generates teacher data. In this embodiment, when a welding image contains a plurality of slags representing impurities generated during welding as a plurality of predetermined events, the teacher data generation unit 102 generates teacher data by extracting the slag with the largest size based on a predetermined criterion from the plurality of slags in the welding image as a feature point (third feature point). Therefore, the teacher data generation unit 102 does not assign position information to slags other than the largest-sized slag, and does not register them in the teacher data 1212.

[0188] Furthermore, in welding control device 26 according to this embodiment, inference unit 26d has the same functions as in the first embodiment, and also uses feature point extraction model 1213 to output position information of one or more feature points and event information based on the distribution of scores of the feature point (third feature point) where an event occurs multiple times but the number of occurrences is unspecified and the number of extractions is single. The specific inference method is the same as the inference process in the first embodiment when there is a first feature point where the occurrence of an event such as a droplet is unspecified and the number of extractions is single.

[0189] Next, the learning process (S103) when the extraction target is a slug will be described. Here, it is assumed that the maximum size slug is extracted as a single feature point as a predetermined criterion.

[0190] FIG. 40 is a flowchart showing an example of the procedure of the learning process when the extraction targets are electrodes, wires, and slugs in the third embodiment.

[0191] In this embodiment, the teacher data generation unit 102 generates actual feature point data from the image information file 1211 of the welding image, and generates teacher data 1212 from the actual feature point data, with the largest size slug as its feature point (S1601). That is, the teacher data generation unit 102 generates teacher data with only the largest size slug among the multiple slugs as its single feature point, and stores the generated teacher data in the storage unit 121.

[0192] Fig. 41 is a diagram showing an example of feature point data showing actual feature points in the third embodiment. In Fig. 41, a plurality of slugs are actual feature points, and slugs 1 to 5 indicate the order of size.

[0193] FIG. 42 is a diagram showing an example of the training data 1212 in the third embodiment. As shown in FIG. 42, the training data 1212 includes an image, and position information of the electrodes, wires, and the maximum size slug in FIG. 41 is added as feature points.

[0194] Next, the learning unit 103 inputs the teacher data 1212 generated by the teacher data generating unit 102 and stored in the storage unit 121 (S801). FIG. 43 is a diagram illustrating an example of the learning process in the third embodiment when the extraction target is slug.

[0195] The subsequent processing is performed in the same manner as in the learning processing in the first embodiment, where the occurrence of an event is unspecified and the number of extractions is single, as explained in Figure 20 using droplets as an example, but with droplets replaced by slags. 40, next, for images in the training data 1212 in which slag is present (i.e., a slag event is present), the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrode, wire, and slag, respectively, as shown in Fig. 43 (S1602). Next, the learning unit 103 inputs the Gaussian distributions for the electrode, wire, and slag to the feature point extraction model 1213, as shown in Fig. 43, and causes the feature point extraction model 1213 to learn (S1603).

[0196] In parallel with the above process, the learning unit 103 generates Gaussian distributions (normal distributions) centered on the electrodes and wires for images without slag (i.e., no slag events) in the teacher data 1212 (S1604), as shown in Fig. 43. Next, the learning unit 103 inputs the Gaussian distributions of the electrodes and wires to the feature point extraction model 1213, as shown in Fig. 43, and causes the feature point extraction model 1213 to learn (S1605).

[0197] Next, the learning unit 103 inputs to the feature point extraction model 1213 that the target image (that is, the image without slug) does not have a slug, and causes the feature point extraction model 1213 to learn (S1606).

[0198] After the learning unit 103 has performed the above process on all images in the training data 1212, it copies the learned feature point extraction model 1213 to the storage device 30 and returns to the caller of the process.

[0199] Next, the inference process (S902) when the extraction target is a slug will be described. Fig. 44 is a flowchart showing an example of the procedure for inference processing when the extraction target is an electrode, wire, or slag in the third embodiment. The inference processing in this embodiment is performed in the same manner as the learning processing in the first embodiment, in which the occurrence of an event is unspecified and the number of extractions is single, as explained in Fig. 23 using droplets as an example, but with droplets replaced by slag.

[0200] FIG. 45 is a diagram showing an example of input information, a processing example, and output information when the extraction target is an electrode, a wire, or a slag in the third embodiment.

[0201] That is, as shown in Fig. 45, the inference unit 26d inputs each welding image to the feature point extraction model 1213. Then, the inference unit 26d uses the feature point extraction model 1213 to create a score distribution of each score for each of the electrode, wire, and slag for each welding image (S1701).

[0202] Next, the inference unit 26d determines the positions of the feature points of the electrodes, wires, and slugs from the score distributions of the electrodes, wires, and slugs generated in S1101, using the feature point extraction model 1213 (S1702).

[0203] Next, the inference unit 26d uses the feature point extraction model 1213 to determine whether or not there is a slug from the score distribution of the slug position (S1703). Next, the inference unit 26d outputs, as output information, position information of each of the determined feature points of the electrode, wire, and slag and the scores at the positions (S1704) using the feature point extraction model 1213. As shown in Fig. 45, output information including the position coordinates and scores of each of the electrode, wire, and slag in the welding image is output, and the inference unit 26d transfers the output information to the control unit 26c.

[0204] As described above, in the information processing device 10 according to this embodiment, the learning unit 103 learns the feature point extraction model 1213 using training data 1212 that includes, among one or more feature points, a feature point (third feature point) in which an event occurs multiple times but the number of occurrences is unspecified and the number of extractions is single.

[0205] Therefore, according to this embodiment, the feature point extraction model 1213 can be properly trained even for events in which the number of occurrences is unspecified and the number of extracted feature points is single, thereby enabling welding to be properly controlled.

[0206] Furthermore, in the information processing device 10 according to this embodiment, when a welding image contains a plurality of predetermined events that occur multiple times but the number of occurrences is unspecified, the teacher data generation unit 102 extracts a single event from the plurality of predetermined events in the welding image based on a predetermined criterion as the feature point (3nth feature point) to generate teacher data 1212. Specifically, when a welding image contains a plurality of slags as the plurality of predetermined events, the teacher data generation unit 102 extracts, from the plurality of slags in the welding image, the slag that is largest in size based on a predetermined criterion as the feature point (third feature point) to generate teacher data 1212.

[0207] That is, according to this embodiment, when there are a plurality of predetermined events that occur multiple times but the number of occurrences is unspecified, a single feature point of the largest size event is selected to generate training data 1212. Therefore, according to this embodiment, training data can be appropriately generated even for an event that occurs multiple times but the number of occurrences is unspecified and that has a single extracted feature point, and the feature point extraction model 1213 can be appropriately trained, thereby enabling welding to be appropriately controlled.

[0208] Furthermore, in the welding control device 26 according to this embodiment, the inference unit 26d uses the feature point extraction model 1213 to output the position information of the feature point and the event information based on the distribution of scores of the feature point (third feature point) among one or more feature points, where an event occurs multiple times but the number of occurrences is unspecified and the number of extractions is single.

[0209] For this reason, according to this embodiment, welding is controlled based on the feature point position information, scores, and event information output using a feature point extraction model 1213 that outputs not only feature point position information but also event information indicating whether or not an event has occurred related to the largest slag in size, which is a feature point (third feature point) that, among one or more feature points, has multiple occurrences but the number of occurrences is unspecified and the number of extracted events is single. Therefore, according to this embodiment, by performing inference using such a feature point extraction model, various feature points can be extracted and welding can be appropriately controlled.

[0210] (Variation) Various modifications of the above embodiment are possible. In the above embodiment, the learning unit 103 obtains a Gaussian distribution (normal distribution) from the positions of multiple feature points in the teacher data 1212, and uses the Gaussian distribution to learn a feature point extraction model. However, the Gaussian distribution is just an example, and a predetermined distribution other than the Gaussian distribution can also be used.

[0211] In the above embodiment and modified example, the information processing device 10 includes the learning unit 103, and the welding control device 26 includes the inference unit 26d, but this is not limiting. For example, the learning unit 103 and the inference unit 26d may be configured to be included in a single device.

[0212] The information processing device 10 and welding control device 26 of the above embodiment and modified example are equipped with a control device such as a CPU, a storage device such as a ROM (Read Only Memory) or RAM, an external storage device such as an HDD, SSD or CD drive, a display device such as a display device, and input devices such as a keyboard or mouse, and have a hardware configuration that utilizes a normal computer.

[0213] Each unit of the information processing device 10 according to the above embodiment and modified examples (acquisition unit 101, teacher data generation unit 102, learning unit 103, and output control unit 104) and each unit of the welding control device 26 (control unit 26c, inference unit 26d) are realized, for example, by one or more processors. For example, each unit may be realized by having a processor such as a CPU execute a program, i.e., by software. Also, each unit may be realized by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each unit may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or two or more of the units.

[0214] The learning program executed by the information processing device 10 according to the above-described embodiment and modified example is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or a Blu-ray (registered trademark) Disc.

[0215] The learning program executed by the information processing device 10 according to the embodiment and the modified example may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The learning program executed by the information processing device 10 according to the embodiment and the modified example may be provided or distributed via a network such as the Internet.

[0216] Furthermore, the learning program executed by the information processing device 10 according to the above embodiment and modified example may be provided by being pre-installed in a ROM or the like.

[0217] The inference program executed by the welding control device 26 according to the above-described embodiment and modified example is provided as a file in an installable or executable format recorded on a computer-readable recording medium such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), or a Blu-ray (registered trademark) Disc.

[0218] The inference program executed by welding control device 26 according to the above embodiment and modified examples may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. The inference program executed by welding control device 26 according to the above embodiment and modified examples may be provided or distributed via a network such as the Internet.

[0219] The inference program executed by the welding control device 26 according to the above embodiment and modified example may be provided by being pre-installed in a ROM or the like.

[0220] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0221] 1. Welding System 10. Information processing equipment 20 Welding equipment 21 head 21a electrode 22 Arm 23 wire 24 Imaging unit 25 Lighting Department 26 Welding control device 26a Power supply section 26b Gas supply unit 26c Control section 26d Reasoning part 30 Storage device 40 PoE Hub 50 PLC 101 Acquisition Department 102 Teacher data generation unit 103 Learning Department 104 Output control section 105 Input section 106 Display section 121 Storage section 1211 Image Information File 1212 Teacher Data 1213 Feature Point Extraction Model

Claims

1. a learning unit that uses training data including one or more welding images of an object to be welded, true values ​​of position information of one or more feature points in the welding images, and event information indicating whether or not an event related to the welding has occurred as a first feature point of which the occurrence of an event is unspecified and the number of extractions is single among the one or more feature points, to learn a feature point extraction model that inputs the welding images and outputs position information of the feature points, an evaluation value indicating the reliability of the position information, and the event information; An information processing device comprising:

2. the learning unit, when the first feature point does not exist in the welding image, uses the training data to learn that the event does not occur in the feature point extraction model; The information processing device according to claim 1 .

3. the learning unit obtains a predetermined distribution for a plurality of feature points from the training data, and learns the feature point extraction model using the predetermined distribution. The information processing device according to claim 2 .

4. The first feature point includes a feature point of any one of a droplet when a droplet falls, a hole when a digging defect occurs, and a hole when a burn-through occurs. The information processing device according to claim 3 .

5. the learning unit obtains, from the teacher data, a predetermined distribution for each of a plurality of second feature points, among the plurality of feature points, where the number of occurrences of an event is unspecified and the number of extractions is multiple, and learns the feature point extraction model using a second distribution obtained by combining the plurality of predetermined distributions. The information processing device according to claim 2 .

6. the plurality of second feature points include feature points of any of slag indicating impurities generated during welding, abnormalities in the weld bead, and foreign matter adhering to the electrode; The information processing device according to claim 5 .

7. the learning unit learns the feature point extraction model using training data including sizes of a plurality of second feature points, the plurality of feature points being extracted in a plurality of instances where an event occurs and the number of occurrences is unspecified; and The information processing device according to claim 2 .

8. the plurality of second characteristic points include a characteristic point of any one of slag indicating impurities generated during welding and foreign matter adhering to the electrode; The information processing device according to claim 7 .

9. the learning unit learns the feature point extraction model using training data including a third feature point, among the one or more feature points, where an event occurs multiple times but the number of occurrences is unspecified and the number of extractions is single. The information processing device according to claim 3 .

10. a teacher data generation unit that, when a plurality of predetermined events, each of which occurs multiple times and the number of occurrences is unspecified, extracts a single event from the plurality of predetermined events in the welding image according to a predetermined criterion as the third feature point, and generates the teacher data; The information processing device according to claim 9 , further comprising:

11. the teacher data generation unit generates teacher data by extracting, as the third feature point, a slag having a maximum size as a predetermined criterion from the plurality of slags in the welding image when the plurality of slags indicating impurities generated during welding are present as the predetermined plurality of phenomena in the welding image; The information processing device according to claim 10.

12. an inference unit that receives as input one or more welding images of an object to be welded, and outputs, using a feature point extraction model that outputs position information of one or more feature points in the welding image, an evaluation value indicating the reliability of the position information, and event information indicating whether or not an event related to the welding has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of which is single, from the welding image, the position information of the feature points, the evaluation value, and the event information; a control unit that controls welding based on the position information of the feature points, the evaluation value, and the event information output by the inference unit; A control device comprising:

13. the inference unit uses the feature point extraction model to output the position information of the feature points and the event information based on a distribution of the evaluation values ​​of the feature points. The control device according to claim 12.

14. the inference unit uses the feature point extraction model to output the position information of the plurality of second feature points and the event information based on a distribution of the evaluation values ​​of a plurality of second feature points, among the plurality of feature points, where an event occurs in a plurality of places but the number of occurrences is unspecified and a plurality of extracted points are included. The control device according to claim 12.

15. The inference unit further outputs sizes of the plurality of second feature points when the event has occurred based on the distribution of the evaluation values, using the feature point extraction model. The control device according to claim 14.

16. the inference unit uses the feature point extraction model to output the position information of the third feature point and the event information based on a distribution of the evaluation values ​​of a third feature point, which is a single extracted feature point and for which an event occurs multiple times but the number of occurrences is unspecified, among the one or more feature points. The control device according to claim 12.

17. An information processing method executed by an information processing device, a step of learning a feature point extraction model that inputs the welding image and outputs the position information of the feature points, an evaluation value indicating the reliability of the position information, and the event information, using training data including one or more welding images of an object to be welded, true values ​​of position information of one or more feature points in the welding image, and event information indicating whether or not an event related to the welding has occurred as a first feature point of which the occurrence of an event is unspecified and the number of extractions is single, among the one or more feature points; An information processing method including:

18. A control method executed by a control device, a step of inputting one or more welding images of an object to be welded, and using a feature point extraction model that outputs position information of one or more feature points in the welding image, an evaluation value indicating the reliability of the position information, and event information indicating whether or not an event related to the welding has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of which is single, and outputting the position information of the feature points, the evaluation value, and the event information from the welding image; controlling welding based on the output position information of the feature points, the evaluation value, and the event information; A control method comprising:

19. A program to be executed by a computer of an information processing device, a step of learning a feature point extraction model that inputs the welding image and outputs the position information of the feature points, an evaluation value indicating the reliability of the position information, and the event information, using training data including one or more welding images of an object to be welded, true values ​​of position information of one or more feature points in the welding image, and event information indicating whether or not an event related to the welding has occurred as a first feature point of which the occurrence of an event is unspecified and the number of extractions is single, among the one or more feature points; A program for causing the computer to execute the above.

20. A program to be executed by a computer of a control device, a step of inputting one or more welding images of an object to be welded, and using a feature point extraction model that outputs position information of one or more feature points in the welding image, an evaluation value indicating the reliability of the position information, and event information indicating whether or not an event related to the welding has occurred as a first feature point among the one or more feature points, the occurrence of which is unspecified and the number of which is single, and outputting the position information of the feature points, the evaluation value, and the event information from the welding image; controlling welding based on the output position information of the feature points, the evaluation value, and the event information; A program for causing the computer to execute the above.

Images