Processing method, computer and storage medium

Abstract

Provided are a processing method, a processing system, and a storage medium that enable easier preparation of a machine learning model and checking of an algorithm. In the processing method of an embodiment, a model selection process and an algorithm selection process are included for execution by a computer. In the model selection process, the computer receives input of a kind of welding and a kind of joint, and selects a machine learning model corresponding to a combination of the kind of welding and the kind of joint from among a plurality of machine learning models. In the algorithm selection process, the computer acquires a first image captured in welding and a first label representing a feature of the welding in the first image, creates a plurality of checking algorithms each containing two or more checking parameters, inputs the first image to the selected machine learning model, calculates, for each checking algorithm, a feature value representing the feature of the welding appearing in the first image using a first output result output from the machine learning model, and selects a certain checking algorithm based on a comparison result of a calculation result of the feature value of each checking algorithm and the first label.

Description

Technical Field

[0001] Embodiments of the present invention relate to processing methods, computers, and storage media. Background Technology

[0002] There are techniques for inspecting welds based on images captured during the welding process. For example, conventional methods extract the molten pool from an image and determine its quality based on this extracted pool. A learned machine learning model is used in the molten pool extraction. In conventional methods, a machine learning model for extracting the molten pool from the image is prepared, and a preferred inspection algorithm is developed when this machine learning model is used. Regarding this method, there is a requirement for techniques that can more easily prepare both the machine learning model and the inspection algorithm.

[0003] Existing technical documents Patent documents Patent Document 1: Japanese Patent No. 6792880 Summary of the Invention

[0004] The technical problem that the invention aims to solve The technical problem to be solved by the embodiments of the present invention is to provide a processing method, processing system and storage medium that can more easily prepare machine learning models and check algorithms.

[0005] Means for solving technical problems In the processing method of this implementation, the computer performs model selection processing and algorithm selection processing. In the model selection processing, the computer receives input of the type of weld and the type of joint. The computer refers to multiple machine learning models that extract regions of the molten pool from images captured during welding, and selects a machine learning model corresponding to the combination of the type of weld and the type of joint. In the algorithm selection processing, the computer obtains a first image captured during welding and a first label representing the features of the weld in the first image. The computer refers to multiple inspection parameters used in welding inspection and creates multiple inspection algorithms, each containing two or more of the inspection parameters. The computer inputs the first image into the selected machine learning model, and using a first output result from the machine learning model, calculates feature values ​​representing the features of the weld reflected in the first image according to each inspection algorithm. Based on a comparison between the calculated feature values ​​of each inspection algorithm and the first label, the computer selects one of the multiple inspection algorithms. Attached Figure Description

[0006] Figure 1 This is a schematic diagram illustrating the processing system of the implementation method.

[0007] Figure 2 This is a flowchart illustrating the processing method of the implementation method.

[0008] Figure 3 This is a table that shows the parameters to be checked.

[0009] Figure 4 This is a table that illustrates the checking algorithm.

[0010] Figure 5 This is a schematic diagram illustrating the processing of a machine learning model.

[0011] Figure 6 This is a table that shows the execution results of the algorithm.

[0012] Figure 7 This is a schematic diagram illustrating a welding system.

[0013] Figure 8 This is a flowchart illustrating the processing method of a first modified example of the implementation method.

[0014] Figure 9 This is a schematic diagram used to illustrate a calculation example of the evaluation function.

[0015] Figure 10 This is a schematic diagram used to illustrate a calculation example of the evaluation function.

[0016] Figure 11 (a) is a graph illustrating the relationship between the number of periods and the evaluation value. Figure 11 (b) is a graph illustrating the change in evaluation value relative to the number of periods.

[0017] Figure 12 This is a flowchart illustrating the processing method of a second variation of the implementation method.

[0018] Figure 13 It is a schematic diagram representing the hardware structure. Detailed Implementation

[0019] Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the ratio of the sizes between parts, etc., may not be the same as in reality. Even when showing the same parts, there may be cases where the dimensions and ratios of each other are represented differently according to the drawings. Furthermore, in this specification and the drawings, the same reference numerals are used for elements that are already described, and detailed descriptions are appropriately omitted.

[0020] Figure 1 This is a schematic diagram illustrating the processing system of the implementation method.

[0021] like Figure 1As shown, the processing system 1 includes a computer 10, an input device 20, a display device 30, and a storage device 40. The computer 10 performs processing for preparing machine learning models and evaluating algorithms. The input device 20 is used by a user to input information into the computer 10. For example, the input device 20 can be a mouse, keyboard, etc. The display device 30 displays information output from the computer 10. For example, the display device 30 can be a monitor or projector. The storage device 40 stores data required for the processing of the computer 10, data obtained through the processing of the computer 10, etc. The computer 10 is connected to the input device 20, the display device 30, and the storage device 40 via wired communication, wireless communication, or a network.

[0022] Computer 10 functions as a model selection unit 11 and an algorithm selection unit 12. Storage device 40 stores multiple pre-prepared machine learning models. These machine learning models are pre-learned to extract regions of the molten pool from images captured during welding. Additionally, storage device 40 stores multiple inspection parameters usable in calculating feature quantities representing the characteristics of the weld. Model selection unit 11 selects one machine learning model from the multiple pre-prepared models. Algorithm selection unit 12 combines the multiple pre-prepared inspection parameters to create multiple inspection algorithms, and selects one of these algorithms.

[0023] Figure 2 This is a flowchart illustrating the processing method of the implementation method.

[0024] Reference Figure 2 The specific implementation method is described below. The processing method of the implementation method mainly includes model selection processing (step S10) and algorithm selection processing (step S20).

[0025] In the model selection process, the type of weld and the type of joint in the weld are input to the model selection unit 11 regarding the weld for which the calculated feature quantity is calculated. The type of weld and the type of joint are input by the user using the input device 20. Alternatively, the computer 10 or other computer may automatically determine the type of weld and the type of joint based on the data used to enable the welding device to perform the welding, and input them to the model selection unit 11. The model selection unit 11 accepts these inputs (step S11). The model selection unit 11 selects a machine learning model from a plurality of machine learning models stored in the storage device 40 that corresponds to the combination of the input weld type and the input joint type (step S12).

[0026] "Types of welding" refers to the classification of welding techniques used to integrate two or more components. For example, types of welding include one or more selected from the group consisting of electric arc welding, coated arc welding, Metal Inert Gas (MIG) welding, Metal Active Gas (MAG) welding, Tungsten Inert Gas (TIG) welding, laser welding, gas welding, plasma welding, and electron beam welding.

[0027] "Types of joints" refers to the classification of the joining methods used when two or more components are joined together by welding. Joints are classified according to their shape and configuration. For example, joint types include one or more selected from the group consisting of lap joints, butt joints, edge joints, T-joints, cross joints, and corner joints.

[0028] When selecting a machine learning model, in addition to using the type of weld and the type of joint, other classifications can also be used. For example, the machine learning model can also be selected based on three classifications: the type of weld, the type of joint, and the shape of the weld. "Weld shape" refers to the classification corresponding to the shape of the part being welded. For example, the weld shape includes one or more selections from the group including bevel welds, fillet welds, plug welds, and groove welds. In this case, the type of weld, the type of joint, and the shape of the weld are input to the model selection unit 11. The model selection unit 11 accepts these inputs. The model selection unit 11 selects a machine learning model from multiple machine learning models that corresponds to the combination of the input weld type, the input joint type, and the input weld shape.

[0029] When a machine learning model is selected by the model selection unit 11, the algorithm selection unit 12 refers to a number of pre-registered check parameters (step S21).

[0030] Figure 3 This is a table that shows the parameters to be checked.

[0031] For example, such as Figure 3 As shown, multiple parameters are pre-registered. Figure 3 In the example shown, parameters related to absolute position, parameters related to relative position, parameters related to shape, and parameters related to approval conditions are registered.

[0032] "Absolute position" refers to the location (coordinates) of a specific point in an image. The position of a specific point is calculated based on the melt pool extracted by a machine learning model. For example, parameters related to absolute position include the centroid, the position with the largest or smallest coordinates, etc.

[0033] "Relative position" refers to the location of a specific point in an image relative to a predetermined reference point. For example, based on the extracted melt pool, the positions of at least two points are calculated according to a pre-set reference. Parameters related to relative position include the angle between the line segment connecting these two points and the predetermined reference line, or the distance between these two points, etc.

[0034] For example, in welding, holes sometimes form in the molten pool. Holes can be caused by bubbles generated in the molten pool, burn-through due to excessive heat input, etc. The location of the hole can also be calculated during welding. Holes can be detected through image processing. Alternatively, machine learning models for hole detection can be used. When a hole is detected, parameters such as the angle between the line segment connecting the centroid of the molten pool and the hole, and the distance between them, can be used as parameters for relative position.

[0035] "Shape" refers to the extracted shape of the molten pool. For example, shape-related parameters include the area of ​​the molten pool, the area ratio relative to a reference area, the diameter of the circumcircle of the molten pool, the diameter of the incircle of the molten pool, and the shape spectrum or aspect ratio. The shape spectrum represents shape information as frequency components. For example, the coordinates (x, y) of each vertex in the outer edge of the molten pool are obtained. The coordinates of each vertex are transformed into polar coordinates centered on the centroid of the molten pool. Through polar coordinate transformation, each coordinate (x, y) is represented by the angle θ relative to the reference line and the distance r from the center. By performing a Fourier transform on the change in distance r relative to angle θ, the intensity and phase shift of each frequency component in the dynamic diameter length distribution r(θ) are calculated as the shape spectrum.

[0036] "Approval criteria" refer to the conditions used to determine whether a result obtained through inspection is correct and approved. For example, parameters related to approval criteria include the number of consecutive judgments, exclusion period, periodicity, inter-frame variation, or moving average. More than one value can be set for each parameter of the approval criteria.

[0037] As an example, the quality of the weld is determined based on the molten pool extracted from a machine learning model. In this case, such as Figure 3 As shown, conditions for comparing the calculation results of at least some parameters are preset. The welding quality of each parameter is determined by comparing the calculation results of each parameter with the comparison conditions. For example, regarding the center of gravity position, if the difference between the extracted center of gravity position of the molten pool and the set value (reference coordinates) is less than 10 pixels, the welding is considered good. The set value is set by the algorithm selection unit 12 when the inspection parameter is selected.

[0038] When determining the quality of welding during inspection, the parameter for approval criteria, the number of consecutive judgments, indicates how many consecutive judgments are required to determine a "good" or "bad" result. If the number of consecutive judgments is 3, and a particular judgment result of "good" or "bad" is obtained 3 times consecutively, then that result is approved.

[0039] The exclusion period indicates the conditions for excluding images from inspection. During the exclusion period, a trigger is preset by the user. Once the trigger conditions are met, images acquired during the exclusion period are not used for inspection. For example, the exclusion period is set to 5 frames (5f), and the trigger is set to generate a hole of a specified size or larger in the molten pool. In this case, if it is determined from the image that a hole of a specified size or larger has been generated in the molten pool, images acquired during the 5-frame period from the generation of the hole are not used for inspection.

[0040] Periodicity refers to the periodicity of images used for inspection when they are extracted periodically. For example, when the periodicity is set to 3 frames, the inspection results are compared with those of every 3 frames. If the inspection results are the same, the result is approved if the number of consecutive judgments exceeds the number set in the "number of consecutive judgments". For example, in welding, luminescence is sometimes generated periodically. It is difficult to obtain an appropriate inspection result from the image when the luminescence occurs. By aligning the periodicity value with the period of luminescence, the timing of the image used for approval of the inspection result is staggered from the timing of the luminescence, thereby obtaining a more appropriate inspection result.

[0041] Inter-frame variation involves the difference between specified inter-frame check values. Regarding inter-frame variation, the change (difference) between the check value obtained from an image at a certain timing and the check value obtained from an image at a timing after a specified frame is calculated. In inter-frame variation, frame differences for these timings and a reference value for the variation are set. Figure 3 In the example shown, the inter-frame variation is set to 2 frames and the difference to 10%. In this case, the inspection result is approved if the change in the check value between a certain timed image and an image two frames later is less than 10%.

[0042] Here, the check value refers to the value obtained by calculating the check parameters based on the extraction results of the molten pool. For example, based on the extraction results of the molten pool, calculate... Figure 3 The parameters shown are related to absolute position, relative position, or shape, from which the check values ​​are obtained.

[0043] The moving average is set when calculating the average of the check values ​​over a specified period. For example, if the moving average is set to 2f, the average of the check values ​​in the images of a given frame and the next frame is calculated. Then, it is determined whether the moving average of the check values ​​meets the comparison criteria set for each check value. Furthermore, the moving average itself is not a parameter used to approve the check results; therefore, it is used in combination with other approval criteria.

[0044] The setting values ​​for each parameter related to the approval conditions are set by the algorithm selection unit 12 when the inspection parameter is selected.

[0045] Algorithm selection unit 12 from Figure 3 One or more inspection parameters can be selected from the multiple inspection parameters shown. When selecting an inspection parameter, the algorithm selection unit 12 appropriately sets... Figure 3 The setting values ​​are shown in the column on the right. The algorithm selection unit 12 generates a check algorithm that includes the selected check parameters (step S22). The algorithm selection unit 12 repeatedly selects check parameters and generates check algorithms, thereby generating multiple check algorithms.

[0046] An inspection algorithm includes one or more parameters selected from multiple parameters related to absolute position, multiple parameters related to relative position, and multiple parameters related to shape, as well as one or more parameters selected from multiple parameters related to approval conditions. Parameters related to absolute position, relative position, or shape, and parameters related to approval conditions are used under AND conditions. When selecting two or more parameters from multiple parameters related to absolute position, multiple parameters related to relative position, and multiple parameters related to shape, these parameters can be used under either AND or OR conditions. Whether these parameters are used under AND or OR conditions can be pre-specified by the user or set by the algorithm selection unit 12. As an example, in determining whether a weld is good or bad during inspection, it is judged as "good" based on one parameter related to absolute position, relative position, or shape, and as "bad" based on other parameters. In this case, if the AND condition is applied, the weld is judged as "bad." If the OR condition is applied, the weld is judged as "good."

[0047] Figure 4 This is a table that illustrates the checking algorithm.

[0048] exist Figure 4 In the example shown, M checking algorithms were created. Each checking algorithm includes one or more checking parameters selected from N checking parameters. The checking parameters can be selected randomly. For example, the number of checking algorithms created and the upper limit of the number of parameters included in each algorithm can be preset. The algorithm selection unit 12 creates the preset number of checking algorithms.

[0049] When creating the algorithm, the algorithm selection unit 12 can also adjust the conditions of each parameter. For example, such as Figure 3 As shown, under certain parameter conditions, the value calculated based on the extracted melt pool is compared with a reference value. An initial value is set for each reference value. When creating the algorithm, the algorithm selection unit 12 can use the initial value as the reference value, or it can change the initial value to use it as the reference value. Therefore, even when multiple algorithms contain common parameters, the conditions used for at least some of the parameters may differ.

[0050] When multiple inspection algorithms are developed, the algorithm selection unit 12 evaluates each inspection algorithm. Specifically, the algorithm selection unit 12 obtains one or more pre-prepared evaluation datasets (step S23). Each evaluation dataset includes an image (first image) and a label (first label). The image is taken in advance when welding of the type selected in step S11 is performed on the joint of the type selected in step S11. The label represents the correct solution for the image. The feature quantity representing the features reflected in the image is used as the label. The algorithm selection unit 12 inputs the images contained in the evaluation dataset into a machine learning model and obtains the output result from the machine learning model (first output result) (step S24).

[0051] Figure 5 This is a schematic diagram illustrating processing based on a machine learning model.

[0052] Figure 5 Image 100 is included in the evaluation dataset. Image 100 was captured during MAG welding. Image 100 shows electrode 101, discharge 102, and molten pool 103. A machine learning model extracts the molten pool from the image. Figure 5 In the diagram, the solid line represents the actual range of the molten pool 103. The dashed line represents the molten pool 103 extracted by the machine learning model.

[0053] The algorithm selection unit 12 uses the output from the machine learning model to execute each inspection algorithm prepared in step S22 (step S25). By executing the inspection algorithms, the extraction results of the melt pool are used to calculate the inspection parameters included in each inspection algorithm, obtaining inspection values. Each inspection algorithm contains more than one inspection parameter. Therefore, more than one inspection value is calculated for each inspection algorithm. For example, the inspection value is a specific numerical value such as the center of gravity position, angle, distance, or area.

[0054] By executing the inspection algorithm, feature quantities representing the characteristics of the weld reflected in the image are calculated. For example, the feature quantity indicates whether the weld is good or bad. When the feature quantity is "0", it indicates that the weld reflected in the image is poor. When the feature quantity is "1", it indicates that the weld reflected in the image is good. The algorithm selection unit 12 uses the inspection value to calculate the feature quantity representing whether the weld is good or bad for each inspection algorithm. When calculating the position or size of the weld, the algorithm selection unit 12 uses the inspection value to calculate the feature quantity representing the position or size for each inspection algorithm.

[0055] The algorithm selection unit 12 compares the calculation results of each checking algorithm with the positive solution data contained in the evaluation dataset (step S26). Based on the comparison results, the algorithm selection unit 12 selects one checking algorithm from multiple checking algorithms (step S27).

[0056] Figure 6 This is a table that shows the execution results of the algorithm.

[0057] Figure 6 Table 110 shows data number 111, positive answer data 112, and calculation results 113. Data number 111 indicates the number of the evaluation dataset. Positive answer data 112 indicates the label (value of the positive answer data) of each evaluation dataset. Calculation results 113 indicate the feature quantities calculated by each inspection algorithm. Figure 6 In the example shown, the label and the feature quantity indicate whether the weld is good or bad.

[0058] For example, in data No.1, the correct answer is "1" (good). According to inspection algorithm No.1, the weld reflected in the image is judged as "0" (bad). On the other hand, according to inspection algorithm No.4, the weld is judged as "1". This means that inspection algorithm No.4 can correctly judge the weld in the image contained in data No.1, but inspection algorithm No.1 cannot correctly judge the weld.

[0059] The algorithm selection unit 12 compares the correct answer rates of each checking algorithm and selects the checking algorithm with the highest correct answer rate. Through the above processing, a machine learning model and a checking algorithm are selected.

[0060] The computer 10 can also display information related to the above-described processing on the display device 30. For example, the computer 10 may display the selected machine learning model, the selected inspection algorithm, Figure 5 The extraction results shown Figure 6 The feature values ​​of each checking algorithm or the positive solution rate of each checking algorithm are displayed on the display device 30.

[0061] After selecting the machine learning model and inspection algorithm, they can be used to perform weld inspections.

[0062] During inspection, the dimensions or location of the weld can also be calculated based on the extracted molten pool. In this case, it is also possible to use... Figure 3 The example shown uses the same check parameters.

[0063] As an example, feature values ​​representing the size of the molten pool are calculated based on the extracted molten pool. The size of the molten pool is calculated based on the difference between the maximum and minimum positions of the extracted molten pool. For example, in the inspection algorithm, the angle and distance of the line segment connecting the centroid of the molten pool to the concave position are used as parameters for relative position. A shape spectrum is used as a parameter for shape. For example, the comparison condition for angle is set to 290–340 degrees. The comparison condition for distance is set to 50–100 pixels. The shape spectrum is set to 2–4. The number of consecutive judgments for the approval condition is set to “3”. These parameters are used under the AND condition. In this case, the molten pool is extracted from each image obtained by capturing the molten pool, and the angle, distance, and shape spectrum are calculated. Images with angles of 290–340 degrees, distances of 50–100 pixels, and shape spectra of 2–4 are extracted from multiple images. The difference between the maximum and minimum positions of the molten pool in the extracted images is calculated as a feature value. The feature value is approved when the difference satisfies the comparison condition three times consecutively.

[0064] Figure 7 This is a schematic diagram illustrating a welding system.

[0065] Figure 7 The welding system 150 shown includes a welding device 151, an imaging device 152, and an inspection device 153. The welding device 151 performs welding. The imaging device 152 captures images of the welding process. The inspection device 153 repeatedly performs inspection processing, acquiring images and calculating feature quantities based on the images.

[0066] During the inspection process, the method used is... Figure 2 The processing shown employs machine learning models and inspection algorithms selected for the process. For example, in the inspection process, the quality of a weld, its location, or its dimensions are calculated as features.

[0067] The inspection device 153 may also function as a detection unit 153a. The detection unit 153a investigates whether there are any anomalies in the extraction results of the melt pool by the machine learning model, and whether there are any anomalies in the inspection values ​​calculated by the inspection algorithm. When an anomaly is detected, the detection unit 153a extracts the image of the detected anomaly from the image captured by the imaging device 152 and saves it to the storage device 154. The storage device 40 may also be used as the storage device 154. In addition, the detection unit 153a sends a notification to the terminal device 155. For example, the notification may contain a message or ID indicating that an anomaly has been detected. The notification may also indicate the necessity of updating the machine learning model.

[0068] In determining whether something is an anomaly, the criteria for whether it is an outlier can be used. For example, test statistics can be used to statistically determine whether something is an outlier.

[0069] As an example, the detection unit 153a calculates the centroid position of the extracted molten pool by checking the execution of the algorithm. The test statistic can be calculated based on a gradient test. In the gradient test, the test statistic G is calculated using the calculated value v, mean x, and variance σ, by G = (vx) / σ. The mean x and variance σ are calculated using values ​​obtained from previous checks. The detection unit 153a calculates the test statistic G for the centroid position of the molten pool. For example, the significance level is 5%, and the sample size is 20. In this case, when the test statistic G exceeds 2.557, the detection unit 153a determines that the value v (centroid position of the molten pool) is an outlier.

[0070] The welding system 150 may also include a learning device 156. The inspection device 153 uses images of detected anomalies to update the machine learning model. In the update, the image of the detected anomaly and annotations for that image are used. The annotations represent the molten pool mapped onto the image.

[0071] During the update, additional learning can be performed on only the images where anomalies were detected. Alternatively, relearning can be performed on all images, including those used in previous learning. After the machine learning model is updated, the inspection device 153 uses the updated machine learning model to perform the inspection.

[0072] In the welding system 150, a single computer can function as both an inspection device 153 and a learning device 156, or different computers can function as both an inspection device 153 and a learning device 156, respectively. Figure 1 The computer 10 shown can also function as an inspection device 153 and a learning device 156.

[0073] The advantages of the implementation method are explained.

[0074] Previously, machine learning models were used to extract the weld pool, and the extracted weld pool was then used for weld inspection. In these methods, the machine learning model was learned based on various conditions such as weld type, joint type, and other welding parameters. Then, the extraction results from the learned model were used to design the optimal inspection algorithm. In other words, in previous methods, after determining the object to be inspected, the machine learning model was trained, and then the inspection algorithm was designed. Therefore, it took a considerable amount of time to prepare both the machine learning model and the inspection algorithm.

[0075] In this implementation, computer 10 performs model selection processing and algorithm selection processing. In the model selection processing, a machine learning model is selected based on inputs of the type of weld and the type of joint. In the algorithm selection processing, multiple inspection algorithms, each containing one or more inspection parameters, are created. Then, based on a comparison between the calculated feature values ​​of each inspection algorithm and the label, a specific inspection algorithm is selected.

[0076] The user selects the type of weld and joint, and prepares images and labels for evaluating the inspection algorithm. Computer 10 selects a machine learning model based on the chosen type and uses the prepared images and labels to create the desired inspection algorithm. According to this implementation, it is easier and faster to prepare the machine learning model for extracting the molten pool and the inspection algorithm using the extraction results.

[0077] like Figure 3 As shown, at least a portion of the multiple inspection algorithms generated by the algorithm selection unit 12 may include thresholds for the inspection parameters. For example, the algorithm selection unit 12 refers to an initial value set for each inspection parameter. The algorithm selection unit 12 sets multiple thresholds based on the initial values, thereby generating two or more inspection algorithms that set different thresholds for a single inspection parameter. Thus, even with the same inspection parameters, algorithms with higher inspection accuracy can be generated.

[0078] After selecting a machine learning model and inspection algorithm, welding and inspection processes for that weld are repeatedly performed. Outliers can also be detected by the detection unit 153a during the inspection process. For example, outliers may arise when the accuracy of the molten pool extracted from the image is low. There is a high probability that the machine learning model for the image used in the inspection process where outliers have been identified has not been learned. The detection unit 153a extracts and saves images that may not have been learned. The saved images can be used to update the machine learning model.

[0079] Furthermore, when the detection unit 153a detects an outlier, it sends a notification to the designated terminal device 155. This allows the necessity of updating the machine learning model to be quickly communicated to the machine learning model administrator, the person in charge of inspection, and others.

[0080] (First variation) Figure 8 This is a flowchart illustrating the processing method of a first modified example of the implementation method.

[0081] Figure 8 The processing method shown is the same as Figure 2 Compared to the processing method shown, steps S13 to S17 are also performed in the model selection process.

[0082] After selecting a machine learning model in step S12, the model selection unit 11 can also determine whether additional learning is needed for the selected machine learning model. For example, the model selection unit 11 acquires an evaluation image (second image) for evaluating the accuracy of the selected machine learning model (step S13). The evaluation image is taken during the process of performing the input welding type for the input joint type.

[0083] The model selection unit 11 inputs the evaluation image into the machine learning model and obtains the output result (second output result) from the machine learning model (step S14). In order to evaluate the accuracy of the machine learning model, the model selection unit 11 uses its output result to calculate an evaluation function (step S15). The model selection unit 11 determines whether the accuracy of the machine learning model is sufficient by comparing the calculated result of the evaluation function with a threshold (step S16).

[0084] For example, the model selection unit 11 calculates the Intersection over Union (IoU). IoU is a value representing the degree of overlap between the region predicted by the machine learning model and the correct answer region. IoU is calculated by dividing the area of ​​the common region between the predicted region and the correct answer region by the area of ​​the combined region of the predicted region and the correct answer region. The closer the IoU is to "1", the higher the accuracy of the machine learning model. The model selection unit 11 calculates IoU as an evaluation value representing the accuracy of the machine learning model. The model selection unit 11 compares the IoU with a threshold to determine the accuracy of the machine learning model.

[0085] Figure 9 This is a schematic diagram used to illustrate a calculation example of the evaluation function.

[0086] Alternatively, the evaluation function can also include values ​​calculated based on the extracted melt pool. Figure 9 Image 100 shown and Figure 5 The same. Image 100 shows electrode 101, discharge 102, and molten pool 103. Figure 9 In the diagram, the solid line represents the actual range of the molten pool 103. The dashed line represents the molten pool 103 extracted by the machine learning model. The actual range of the molten pool 103 is taught in advance.

[0087] exist Figure 9In the example shown, the joining is performed along the weld line 104. As an example, the model selection unit 11 calculates the position (X1, Y1) of the centroid C1 of the actual molten pool 103. The model selection unit 11 generates a straight line in a direction (first direction) passing through the centroid C1 and parallel to the weld line 104. The model selection unit 11 calculates the length L1 of the actual molten pool 103 along this straight line. Additionally, the model selection unit 11 generates a straight line in a direction (second direction) passing through the centroid C1 and perpendicular to the weld line 104. The model selection unit 11 calculates the width W1 of the actual molten pool 103 along this straight line.

[0088] The welding line 104 can be detected from an image or calculated based on welding-related data. For example, when the welding torch of the welding apparatus is welding a component in a straight line, the welding line 104 can be calculated based on the position and direction of movement of the torch after the positioning operation.

[0089] Next, the model selection unit 11 calculates the position (X2, Y2) of the centroid C2 of the molten pool 103 extracted by the machine learning model. The model selection unit 11 generates a straight line passing through the centroid C2 and parallel to the weld line 104. The model selection unit 11 calculates the length L2 of the extracted molten pool 103 along this straight line. Additionally, the model selection unit 11 generates a straight line passing through the centroid C2 and perpendicular to the weld line 104. The model selection unit 11 calculates the width W2 of the extracted molten pool 103 along this straight line.

[0090] The model selection unit 11 calculates an evaluation function that includes the actual dimensions (length L1 and width W1) of the molten pool 103 and the extracted dimensions (length L2 and width W2) of the molten pool 103. As an example, the following evaluation function is calculated. The model selection unit 11 compares the calculated evaluation value Loss with a threshold to determine the accuracy of the machine learning model.

[0091] Loss=(|W1-W2| / W1+|L1-L2| / L1) / 2 Figure 10 This is a schematic diagram illustrating another calculation example of the evaluation function.

[0092] Figure 10 Image 200 shown is a photograph taken with Figure 5 The same object was obtained. Image 200 shows electrode 201, discharge 202, and molten pool 203. Figure 10 In the diagram, the solid line represents the actual range of the molten pool 203. The dashed line represents the molten pool 203 extracted by the machine learning model. As another example of calculating the evaluation function, an evaluation function that includes the maximum distance between the molten pool extracted by the machine learning model and the actual molten pool 203 can also be calculated. Specifically, as... Figure 10As shown, the model selection unit 11 calculates the position (X1, Y1) of the centroid C1 of the actual molten pool 203. The model selection unit 11 sets a baseline SL that passes through the centroid C1 and is parallel to the X direction. The model selection unit 11 calculates the distance d between the contour of the actual molten pool 203 and the extracted contour of the molten pool 203 in a direction with an angle θ relative to the baseline SL. The model selection unit 11 calculates the distance d at each angle θ while varying the angle θ at predetermined scale intervals (e.g., 1 degree). The model selection unit 11 uses the maximum distance D among the calculated distances d as the evaluation value Loss. The model selection unit 11 compares the calculated evaluation value Loss with a threshold to determine the accuracy of the machine learning model.

[0093] If the accuracy of the machine learning model is deemed insufficient, the model selection unit 11 performs additional learning on the machine learning model selected in step S12 (step S17). In this additional learning, multiple training datasets prepared for the additional learning are used. Each training dataset includes an image captured during welding and annotations representing the extent of the molten pool in that image.

[0094] In supplemental learning, the model selection unit 11 can automatically set one or more options from the group including the training dataset used, the number of epochs, and the evaluation function. "Number of epochs" refers to the number of times the entire dataset used for learning is input into the model and learned. For example, when using 100 training datasets, using all of them once for learning is "one epoch".

[0095] For example, the model selection unit 11 sequentially inputs multiple training datasets into the machine learning model selected in step S12. Using the output from the machine learning model and the annotations included in the training datasets, an evaluation function is calculated. Based on the calculation result of the evaluation function, the model selection unit 11 extracts a portion of the multiple training datasets. For example, the model selection unit 11 extracts a predetermined number of training datasets in ascending order of evaluation value Loss. Alternatively, the model selection unit 11 may divide the multiple training datasets into multiple classes based on the evaluation value Loss. The model selection unit 11 extracts a portion of the training dataset from each class. In this case, the lower the evaluation value Loss of a class, the more training dataset can be extracted from that class.

[0096] In supplementary learning, the model selection unit 11 may repeatedly calculate the evaluation function to determine the number of periods. For example, the model selection unit 11 calculates the change in the evaluation value Loss relative to the number of periods (ΔLoss / ΔEpoch). The model selection unit 11 determines that supplementary learning is complete when the change (ΔLoss / ΔEpoch) is below a threshold for a predetermined number of periods.

[0097] Figure 11(a) is a graph illustrating the relationship between the number of periods and the evaluation value. Figure 11 (b) is a graph illustrating the change in evaluation value relative to the number of periods.

[0098] exist Figure 11 In (a), the horizontal axis represents the number of periods (Epochs). The vertical axis represents the evaluation value (Loss) calculated using the evaluation function. Figure 11 In (b), the horizontal axis represents the number of periods (Epoch). The vertical axis represents the value obtained by differentiating the evaluation value with respect to the number of periods (ΔLoss / ΔEpoch). As an example, a condition is set for the absolute value of the variation (ΔLoss / ΔEpoch) over 5 periods to be less than "0.1". As another example, the second derivative or moving average of the evaluation value Loss relative to the number of periods (Epoch) can also be calculated and compared with a threshold.

[0099] According to variations of the implementation method, additional learning is performed as needed. Therefore, it is possible to prepare a machine learning model with sufficient accuracy.

[0100] Furthermore, according to a variation, the computer 10 automatically determines one or more of the following: the data used for learning, the number of periods, and the evaluation function. Therefore, the burden on the user requiring additional learning can be reduced.

[0101] (Second variation) Figure 12 This is a flowchart illustrating the processing method of a second variation of the implementation method.

[0102] exist Figure 12 In the processing method shown, with Figure 2 Compared to the processing method shown, step S28 is further performed in the algorithm selection process.

[0103] exist Figure 12 In the processing method shown, steps S22 to S26 are repeated. Step S28 is executed after step S26. In step S28, it is determined whether the termination condition for repeated execution is sufficient. For example, the termination condition may be set as the number of repetitions to a predetermined number or the correct solution rate of a certain checking algorithm exceeding a predetermined threshold.

[0104] If the termination condition is deemed insufficient, the algorithm selection unit 12 executes step S22 again. In this second step S22, a different checking algorithm is created than before.

[0105] In step S22, the correctness rate of previously developed checking algorithms can also be referenced. For example, the algorithm selection unit 12 references a checking algorithm that has achieved the highest correctness rate in the past. The algorithm selection unit 12 generates one or more new checking algorithms by changing at least one checking parameter included in the referenced checking algorithm. That is, the algorithm selection unit 12 searches for a checking algorithm with a higher correctness rate while referring to the previously obtained correctness rate.

[0106] The algorithm selection unit 12 can also adjust the checking algorithm while referring to the correct answer rates of multiple similar checking algorithms. For example, in the past, two or more checking algorithms were created with different thresholds set for a checking parameter, and the correct answer rates of each checking algorithm were obtained. In this case, the correlation between the threshold and the correct answer rate is calculated. The algorithm selection unit 12 searches for a checking algorithm based on this correlation.

[0107] Figure 13 It is a schematic diagram representing the hardware structure.

[0108] As a computer 10, inspection device 153, and learning device 156, for example using Figure 13 The computer 90 shown includes a processing circuit 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0109] ROM 92 stores programs that control the operation of computer 90. ROM 92 stores the programs required for computer 90 to perform the aforementioned processes. RAM 93 functions as a storage area for expanding the programs stored in ROM 92.

[0110] The processing circuit 91 includes a computing device such as a CPU or GPU. The processing circuit 91 uses RAM 93 as its working memory and executes a program stored in at least one of the ROM 92 or storage device 94. During program execution, the processing circuit 91 controls various structures via the system bus 98 to perform various processes.

[0111] Storage device 94 stores the data required for executing the program and the data obtained by executing the program.

[0112] The input interface (I / F) 95 connects the computer 90 to an input device. The input I / F 95 may be a serial bus interface such as USB. The processing circuitry 91 can read various data from the input device via the input I / F 95.

[0113] Output interface (I / F) 96 connects computer 90 to an output device. Output I / F 96 is, for example, an image output interface such as Digital Visual Interface (DVI) or High-Definition Multimedia Interface (HDMI). Processing circuitry 91 can send data to the output device via output I / F 96, causing the output device to display an image.

[0114] The communication interface (I / F) 97 connects an external server to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The processing circuitry 91 can read various data from the server via the communication I / F 97.

[0115] Storage device 94 includes one or more selected from hard disk drives (HDDs) and solid-state drives (SSDs). Input devices may be one or more selected from mice, keyboards, microphones (voice input), and touchpads. Output devices may be one or more selected from monitors, projectors, printers, and speakers. Devices that function as both input and output devices, such as touchpads, may also be used.

[0116] The processing of the various data mentioned above can also be recorded as programs that can be executed by a computer on a disk (floppy disk and hard disk, etc.), optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), semiconductor memory, or other non-transitory computer-readable storage medium.

[0117] For example, data on a recording medium is read by a computer (or embedded system). The recording format (storage format) on the recording medium is arbitrary. For example, a computer reads a program from a recording medium, causing the CPU to execute instructions based on that program. The retrieval (or reading) of a program by a computer can also be done over a network.

[0118] The embodiments of the present invention include the following features.

[0119] (Feature 1) A processing method includes model selection processing and algorithm selection processing for computer execution. In the model selection process, the computer performs the following steps: Accepts input of welding types and joint types; and Referring to multiple machine learning models that extract regions of the molten pool from images captured during welding, a machine learning model corresponding to the combination of the type of weld and the type of joint is selected. In the algorithm selection process, the computer performs the following steps: Acquire a first image captured during welding and a first label representing the features of the welding in the first image; Based on the multiple inspection parameters used in welding inspection, multiple inspection algorithms are created, each containing two or more of the aforementioned inspection parameters; Input the first image into the selected machine learning model, and using the first output from the machine learning model, calculate feature values ​​representing the features of the weld reflected in the first image according to each inspection algorithm; and Based on the comparison between the calculation result of the feature value of each of the inspection algorithms and the first label, one of the inspection algorithms is selected from the plurality of inspection algorithms.

[0120] (Feature 2) According to the processing method of feature 1, at least a portion of the inspection algorithm includes the more than one inspection parameter and a threshold for each of the inspection parameters.

[0121] (Feature 3) According to the processing method described in feature 2, wherein, The computer performs the following steps: Refer to the initial value set for each of the aforementioned inspection parameters; and Based on the initial value, multiple thresholds are set, thereby creating two or more inspection algorithms that set different thresholds for a single inspection parameter.

[0122] (Feature 4) The processing method according to any one of features 1 to 3, wherein, In the model selection process, the computer performs the following steps: The second image captured during welding is input into the selected machine learning model. Based on the second output from the machine learning model, it is determined whether additional learning is needed for the selected machine learning model. If it is determined that additional learning is needed, additional learning is performed on the selected machine learning model.

[0123] (Feature 5) According to the processing method described in feature 4, the computer decides to select one or more from a group including the data used for learning, the number of periods, and the evaluation function to perform the additional learning.

[0124] (Feature 6) According to the processing method described in feature 4 or 5, wherein, In the additional learning, the computer performs the following steps: Extract the molten pool from the second image; The dimensions of the extracted molten pool are calculated in a first direction parallel to the weld line and a second direction perpendicular to the first direction; and The evaluation function is calculated using the dimensions of the actual molten pool in the first and second directions in the second image and the extracted dimensions of the molten pool.

[0125] (Feature 7) According to the processing method described in feature 4 or 5, wherein, In the additional learning, the computer performs the following steps: Extract the molten pool from the second image; and The evaluation function is calculated using the maximum distance between the actual molten pool contour in the second image and the extracted molten pool contour.

[0126] (Feature 8) According to any one of features 1 to 7, the processing method wherein the computer repeatedly performs an inspection process to calculate feature values ​​using the selected machine learning model and the selected inspection algorithm, the feature values ​​representing the characteristics of the weld reflected in an image captured during the welding process.

[0127] (Feature 9) According to the processing method described in feature 8, wherein, The computer performs the following steps in the inspection process: Calculate the two or more inspection parameters included in the selected inspection algorithm and obtain two or more inspection values; Determine whether at least one of two or more of the stated check values ​​is an outlier; and If at least one of two or more of the check values ​​is an outlier, the image used in the check process that obtained the outlier is extracted.

[0128] (Feature 10) According to the processing method described in feature 8, wherein, The computer performs the following steps in the inspection process: Calculate the two or more inspection parameters included in the selected inspection algorithm and obtain two or more inspection values; Determine whether at least one of two or more of the stated check values ​​is an outlier; and If at least one of two or more of the checked values ​​is an outlier, a notification is sent to the designated terminal device.

[0129] (Feature 11) According to the processing method described in feature 9, the computer uses the extracted image to further learn the selected machine learning model.

[0130] (Feature 12) A computer that executes the processing method described in any one of features 1 to 11.

[0131] (Feature 13) A program that causes a computer to execute the processing method described in any one of features 1 to 11.

[0132] (Feature 14) A storage medium storing the program described in feature 13.

[0133] Based on the embodiments described above, a processing method, computer, program, and storage medium are provided that can more easily and in a shorter time prepare a machine learning model for extracting the melt pool and a checking algorithm using the extraction results.

[0134] In this specification, "or" means "at least one or more" of the items listed in the article can be used.

[0135] Several embodiments of the present invention have been described, but these embodiments are merely illustrative and are not intended to limit the scope of the invention. These new embodiments can be implemented in various other ways, and various omissions, substitutions, modifications, etc., can be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the invention as described in the claims and its equivalents. Furthermore, the described embodiments can be combined with each other for implementation.

Claims

1. A processing method, comprising model selection processing and algorithm selection processing for computer execution. In the model selection process, the computer performs the following steps: Accepts input of welding types and joint types; and Referring to multiple machine learning models that extract regions of the molten pool from images captured during welding, a machine learning model corresponding to the combination of the type of weld and the type of joint is selected. In the algorithm selection process, the computer performs the following steps: Acquire a first image captured during welding and a first label representing the features of the welding in the first image; Based on the multiple inspection parameters used in welding inspection, multiple inspection algorithms are created, each containing two or more of the aforementioned inspection parameters; Input the first image into the selected machine learning model, and using the first output from the machine learning model, calculate feature values ​​representing the features of the weld reflected in the first image according to each inspection algorithm; and Based on the comparison between the calculation result of the feature value of each of the inspection algorithms and the first label, one of the inspection algorithms is selected from the plurality of inspection algorithms.

2. The processing method according to claim 1, wherein, At least a portion of the inspection algorithm includes one or more inspection parameters and thresholds for each of the inspection parameters.

3. The processing method according to claim 2, wherein, The computer performs the following steps: Refer to the initial value set for each of the aforementioned inspection parameters; and Based on the initial value, multiple thresholds are set, thereby creating two or more inspection algorithms that set different thresholds for a single inspection parameter.

4. The processing method according to claim 1, wherein, In the model selection process, the computer performs the following steps: The second image captured during welding is input into the selected machine learning model, and based on the second output result from the machine learning model, it is determined whether additional learning is needed for the selected machine learning model; as well as If it is determined that additional learning is needed, additional learning is performed on the selected machine learning model.

5. The processing method according to claim 4, wherein, The computer decides to select one or more from a group including the data used for learning, the number of periods, and the evaluation function to perform the additional learning.

6. The processing method according to claim 4, wherein, In the additional learning, the computer performs the following steps: Extract the molten pool from the second image; The dimensions of the extracted molten pool are calculated in a first direction parallel to the weld line and in a second direction perpendicular to the first direction. as well as The evaluation function is calculated using the dimensions of the actual molten pool in the first and second directions in the second image and the extracted dimensions of the molten pool.

7. The processing method according to claim 4, wherein, In the additional learning, the computer performs the following steps: Extract the molten pool from the second image; and The evaluation function is calculated using the maximum distance between the actual molten pool contour in the second image and the extracted molten pool contour.

8. The processing method according to claim 1, wherein, The computer uses the selected machine learning model and the selected inspection algorithm to repeatedly perform inspection processing to calculate feature values ​​that represent the characteristics of the weld reflected in the images captured during the welding process.

9. The processing method according to claim 8, wherein, The computer performs the following steps in the inspection process: Two or more check values ​​are obtained by calculating the two or more check parameters included in the selected check algorithm; Determine whether at least one of two or more of the stated check values ​​is an outlier; and If at least one of two or more of the check values ​​is an outlier, the image used in the check process that obtained the outlier is extracted.

10. The processing method according to claim 8, wherein, The computer performs the following steps in the inspection process: Two or more check values ​​are obtained by calculating the two or more check parameters included in the selected check algorithm; Determine whether at least one of the two or more check values ​​is an outlier; If at least one of the two or more check values ​​is an outlier, a notification is sent to the designated terminal device.

11. The processing method according to claim 9, wherein, The computer uses the extracted images to further learn the selected machine learning model.

12. A computer that executes the processing method according to any one of claims 1 to 11.

13. A storage medium storing a program that causes a computer to execute the processing method according to any one of claims 1 to 11.

Images