Method, apparatus, medium and program product for identifying molten pool profile information

Abstract

The invention aims to provide a method and equipment for identifying contour information of a molten pool, a medium and a program product. The method comprises the following steps: acquiring image information of the molten pool; based on the molten pool image information, corresponding molten pool contour information is determined through a trained molten pool contour recognition model, the molten pool contour recognition model comprises a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour tracking model is used for positioning the area where a molten pool is located, and the molten pool contour detection model is used for detecting the molten pool contour. And the molten pool contour detection model is used for identifying the contour of the molten pool in an area where the positioned molten pool is located. The area where the molten pool is located is rapidly positioned through the molten pool contour tracking model, and then the molten pool contour detection model is used for further performing more detailed molten pool contour detection in the target area, so that the anti-interference capability to noise is improved; the image processing rate can be further improved while segmentation errors caused by backgrounds or other interference objects are avoided.

Description

Technical Field

[0001] This application relates to the field of welding, and particularly to a technique for identifying the contour information of a molten pool. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, the demand for real-time monitoring and evaluation of welding quality is increasing day by day. Industry research has found that many contour features of the molten pool can reflect the penetration and formation of the weld seam. Accurately extracting the contour information of the molten pool image is of great significance for welding quality monitoring and evaluation. In recent years, the introduction of deep learning models, especially various convolutional neural networks that perform well in the field of image processing (such as Mask RCNN, Deeplabv3+, etc.), has also significantly improved the contour detection technology. However, in the actual application of molten pool contour detection, these methods are still limited by complex and variable welding conditions. For example, in the face of interference in the molten pool image such as arc light changes, relative camera viewing angle changes, complex backgrounds, and smoke interference, it may cause the model to be unable to accurately distinguish the molten pool from the background, resulting in misidentification. Especially when the molten pool contour is relatively small, the recognition accuracy will decrease significantly. Summary of the Invention

[0003] An object of this application is to provide a method, device, medium, and program product for identifying the contour information of a molten pool.

[0004] According to one aspect of this application, there is provided a method for identifying the contour information of a molten pool, the method including:

[0005] Obtain the molten pool image information;

[0006] Based on the molten pool image information, use the trained molten pool contour recognition model to determine the corresponding molten pool contour information, where the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour within the located area where the molten pool is located.

[0007] According to one aspect of this application, there is provided a computer device for identifying the contour information of a molten pool, including a memory, a processor, and a computer program stored on the memory, characterized in that the processor executes the computer program to implement the steps of any of the above methods.

[0008] According to one aspect of this application, there is provided a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0009] According to one aspect of the present application, there is provided a computer program product including a computer program, characterized in that when the computer program is executed by a processor, it implements the steps of any of the above-mentioned methods.

[0010] According to one aspect of the present application, there is provided a device for identifying molten pool contour information, the device comprising:

[0011] a module for acquiring molten pool image information;

[0012] a second module for determining corresponding molten pool contour information based on the molten pool image information by using a trained molten pool contour recognition model, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour tracking model is used for locating the area where the molten pool is located, and the molten pool contour detection model is used for identifying the molten pool contour within the located area where the molten pool is located.

[0013] Compared with the prior art, the present application acquires molten pool image information; based on the molten pool image information, uses a trained molten pool contour recognition model to determine corresponding molten pool contour information, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour tracking model is used for locating the area where the molten pool is located, and the molten pool contour detection model is used for identifying the molten pool contour within the located area where the molten pool is located. First, the molten pool contour tracking model quickly locates the area where the molten pool is located, and then the molten pool contour detection model further performs more detailed molten pool contour detection within the target area, improving the anti-interference ability to noise, avoiding segmentation errors caused by the background or other interfering objects, and further improving the image processing speed at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0015] Figure 1 A flowchart of a method for identifying molten pool contour information according to an embodiment of the present application is shown;

[0016] Figure 2 A schematic diagram of the annotation result of the molten pool area in a molten pool sample image according to an embodiment of the present application is shown;

[0017] Figure 3 A schematic diagram of the annotation result of the molten pool contour in a molten pool sample image according to an embodiment of the present application is shown;

[0018] Figure 4 A schematic diagram of a double ellipsoid model according to an embodiment of the present application is shown;

[0019] Figure 5 The structural diagram of a device for identifying the information of the molten pool contour according to an embodiment of the present application is shown;

[0020] Figure 6 An exemplary system that can be used to implement the various embodiments described in the present application is shown.

[0021] The same or similar reference numerals in the drawings represent the same or similar components. Detailed implementation manners

[0022] The present application will be further described in detail below with reference to the drawings.

[0023] In a typical configuration of the present application, the terminal, the devices of the service network, and the trusted party all include one or more processors (e.g., a Central Processing Unit (CPU)), an input / output interface, a network interface, and a memory.

[0024] The memory may include non-permanent memory in the computer-readable medium, random access memory (RAM), and / or non-volatile memory in the form of, for example, read-only memory (ROM) or flash memory. The memory is an example of the computer-readable medium.

[0025] Computer-readable media includes both permanent and non-permanent, removable and non-removable media implemented by any method or technology for information storage. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, Phase-Change Memory (PCM), Programmable Random Access Memory (PRAM), Static Random-Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read-Only Memory (ROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technologies, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information accessible by a computing device.

[0026] The devices referred to in this application include, but are not limited to, user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include, but are not limited to, any mobile electronic product that can perform human-computer interaction with users (such as human-computer interaction through a touchpad), such as smart phones, tablets, etc. The mobile electronic products can adopt any operating system, such as the Android operating system, the iOS operating system, etc. Among them, the network device includes an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc. The network device includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud composed of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing. Among them, cloud computing is a type of distributed computing, consisting of a virtual supercomputer formed by a group of loosely coupled computer sets. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network (Ad Hoc network), etc. Preferably, the device can also be a program running on the user device, the network device, or a device formed by integrating the user device and the network device, the network device, the touch terminal, or the network device and the touch terminal through a network.

[0027] Of course, those skilled in the art should understand that the above devices are only examples, and other existing or future devices that are applicable to this application should also be included within the protection scope of this application and are hereby incorporated by reference.

[0028] In the description of this application, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0029] Figure 2The figure shows a flowchart of a method for identifying molten pool contour information according to an embodiment of the present application. The method includes: step S11 and step S12. In step S11, device 1 acquires molten pool image information; in step S12, device 1 determines corresponding molten pool contour information based on the molten pool image information by using a trained molten pool contour recognition model. The molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour within the located area where the molten pool is located.

[0030] In step S11, device 1 acquires molten pool image information. In some embodiments, device 1 includes, but is not limited to, a user device or a network device with information processing or computing capabilities, such as a tablet computer, a computer, a server, etc. Device 1 acquires molten pool image information in real time from an industrial camera or other image sensors installed at the welding site. By identifying the molten pool image information, accurate molten pool contour information can be obtained in real time, so as to facilitate quality diagnosis and adaptive control of the welding process in combination with the molten pool contour information.

[0031] In step S12, device 1 determines corresponding molten pool contour information based on the molten pool image information by using a trained molten pool contour recognition model. The molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour within the located area where the molten pool is located. Before using the molten pool contour detection model to identify the molten pool contour, first use the molten pool contour tracking model to locate the area where the molten pool is located, so as to effectively filter background noise and reduce false detection. Then, the molten pool contour detection model takes the detection result of the molten pool contour tracking model as the model input and only needs to perform detailed detection within the area where the molten pool is located to determine the molten pool contour. By combining the molten pool contour tracking model and the molten pool contour detection model, while avoiding segmentation errors caused by the background or other interfering objects and enhancing the anti-interference ability of the model, the image processing rate can be further improved.

[0032] In some embodiments, step S12 includes: device 1 determines molten pool area information based on the molten pool image information by using the molten pool contour tracking model; and determines molten pool contour information based on the molten pool area information by using the molten pool contour detection model. For example, the molten pool contour tracking model is used for target detection, and filters out irrelevant background information in the image through the molten pool contour tracking model to quickly locate the molten pool area information. Then, the molten pool area information is input into the molten pool contour detection model, and the molten pool contour detection model is used to perform detailed segmentation and identify the molten pool contour within the molten pool area.

[0033] In some embodiments, the method further includes: step S13 (not shown), based on the molten pool contour tracking sample data set, training a molten pool contour tracking model using a target detection model, wherein the molten pool contour tracking sample data set includes first molten pool sample image information and molten pool region annotation information corresponding to the first molten pool sample image information; step S14 (not shown), based on the molten pool contour detection sample data set, combining the molten pool contour tracking model, and training a molten pool contour detection model using a semantic segmentation model, wherein the molten pool contour detection model takes the output of the molten pool contour tracking model as input, and the molten pool contour detection sample data set includes second molten pool sample image information and molten pool contour annotation information corresponding to the second molten pool sample image information.

[0034] For example, select a suitable target detection model (for example, all YOLO series models above YOLOv8), and iteratively train based on the molten pool contour tracking sample data set to obtain a molten pool contour tracking model. Input the second molten pool sample image information into the trained molten pool contour tracking model to determine the corresponding molten pool region information. Input this molten pool region information into the corresponding semantic segmentation model (for example, U-net model (a U-shaped structure semantic segmentation model composed of an encoder and a decoder), etc.), and iteratively train in combination with the molten pool contour detection sample data set to obtain the final molten pool contour detection model.

[0035] In some embodiments, referring to Figure 2 , Figure 3 the schematic diagram of the annotation result of the molten pool sample image shown, corresponding annotation tools (for example, LabelMe (https: / / github.com / wkentaro / labelme), VIA (VGGImage Annotator, https: / / www.robots.ox.ac.uk / ~vgg / software / via / ), etc.) can be used to frame the area where the molten pool is located in the first molten pool sample image information with a rectangular box. The area inside the box is the molten pool area, and the area outside the box is the background area (refer to Figure 2 ), and accordingly, obtain the molten pool region annotation information; use a curve to depict the edge of the molten pool in the second molten pool sample image information. The area inside the curve frame is the molten pool, and the area outside the curve frame is the background (refer to Figure 3 ), and accordingly, obtain the molten pool contour annotation information. In some embodiments, corresponding configuration files can be created respectively based on the molten pool contour tracking sample data set and the molten pool contour detection sample data set for the corresponding model training to standardize and simplify the model training process. Taking the creation of a YAML format configuration file to describe the molten pool contour tracking sample data set as an example, its corresponding YAML file can be set as:

[0036] path:# Path

[0037] train: / path / to / data / train # Address of the training dataset for weld pool contour tracking

[0038] val: / path / to / data / val # Address of the validation dataset for weld pool contour tracking

[0039] test: / path / to / data / test # Address of the test dataset for weld pool contour tracking

[0040] names: # Class names

[0041] 0: background # Background

[0042] 1: weldpool # Weld pool

[0043] In some embodiments, a visualization tool (e.g., TensorBoard, etc.) can be used to monitor the training processes of the weld pool contour tracking model and the weld pool contour detection model, so as to help the user understand the learning process of the model and debug the training of the model.

[0044] In some embodiments, step S13 includes: based on the weld pool contour tracking sample dataset, combined with a first loss function, using an object detection model for iterative training to obtain a weld pool contour tracking model, where the first loss function includes a linear combination of a cross-entropy loss function and an intersection over union loss function.

[0045] For example, during the training process of the weld pool contour tracking model, based on the weld pool contour tracking sample dataset, the difference between the weld pool region information predicted by the weld pool contour tracking model and the true weld pool region annotation information is calculated in combination with the first loss function, and then the gradient of the loss function is calculated, so as to perform optimization iteration of the weld pool contour tracking model to obtain the final weld pool contour tracking model. The cross-entropy loss function (Cross-Entropy Loss) is used to measure the difference between the probability distribution predicted by the model and the true distribution. The smaller the cross-entropy loss value, the closer the predicted probability of the model is to the true label, that is, the better the prediction effect of the model. The formula of the cross-entropy loss function can be expressed as:

[0046]

[0047] where C is the number of classes, y i is the true label, is the probability predicted by the model. The Intersection over Union Loss (IoU Loss) is used to measure the overlapping degree between the predicted region and the ground truth region. IoU (Intersection over Union) is defined as the ratio of the area of the intersection of the predicted region and the ground truth region to the area of their union:

[0048]

[0049] where A is the predicted region and B is the ground truth region. To make IoU applicable for optimization, the IoU loss is defined as: L IoU = 1 - IoU. The smaller the IoU loss value, the better the overlap between the region predicted by the model and the ground truth region, that is, the better the prediction effect of the model. A linear combination of the cross-entropy loss function and the Intersection over Union loss function can be performed with a set weight ratio based on actual requirements. In some embodiments, during the training process of the molten pool contour tracking model, corresponding evaluation metrics (such as Mean Average Precision (mAP), Precision, Recall, F1-score, etc.) can be used to quantify the performance of the model. By combining the calculation results of the evaluation metrics and using the method of controlling variables, the hyperparameters (such as learning rate, optimizer, regularization parameter, etc.) of the molten pool contour tracking model can be adjusted, and techniques such as grid search and random search can be used to systematically explore the hyperparameter space to find the optimal combination of hyperparameters, thereby obtaining the optimal molten pool contour tracking model.

[0050] In some embodiments, the step S14 includes: based on the second molten pool sample image information, using the molten pool contour tracking model to determine the target molten pool region information; based on the target molten pool region information and the molten pool contour detection sample data set, combined with the second loss function, using a semantic segmentation model to train and obtain a molten pool contour detection model, where the second loss function includes a linear combination of the Dice similarity loss function and the cross-entropy loss function.

[0051] For example, the second molten pool sample image information is input into the trained molten pool contour tracking model to determine the target molten pool area information. During the training process of the molten pool contour detection model, the target molten pool area information is used as the input to predict the molten pool contour information. Based on the second loss function, the difference between the predicted molten pool contour information and the true molten pool contour annotation information is calculated, and then the gradient of the loss function is calculated, so as to optimize and iterate the molten pool contour detection model to obtain the final molten pool contour tracking model. The Dice similarity loss function (Dice Loss) is based on the Dice coefficient and is used to measure the similarity between the predicted region and the true region. The Dice loss is sensitive to small targets or class imbalance situations and can well optimize the segmentation effect of small-area targets. The Dice coefficient is usually defined as: Dice = 2×|A∩B| / (|A|+|B|) = 2×TP / (2×TP+FP+FN). Among them, TP (True Positive) is the correctly predicted positive sample; FP is the false positive (False Positive), that is, the number of samples that the model wrongly predicts as the positive class; FN is the false negative (False Negative), that is, the number of positive class samples that the model wrongly predicts as the negative class. The Dice similarity loss function is defined as: Dice Loss = 1 - Dice. The smaller the Dice Loss value, the higher the similarity between the model prediction region and the true region, that is, the better the prediction effect of the model. The cross-entropy loss function is the same as that described in the foregoing step S13 and will not be elaborated here and is included herein by reference. Similarly, based on actual requirements, a linear combination of the Dice similarity loss function and the cross-entropy loss function can be performed with a set weight ratio. In some embodiments, various data augmentation methods (for example, rotation, scaling, flipping, cropping, brightness and contrast adjustment, noise addition, random occlusion, etc.) can also be used to process the sample images, expand the training sample dataset, and improve the detection ability and adaptability of the model to the molten pool contour.

[0052] In some embodiments, the method further includes: step S15 (not shown), where device 1 performs anomaly detection based on the molten pool profile information to obtain welding detection result information. For example, after device 1 accurately obtains the molten pool profile information in real time using the aforementioned trained molten pool profile recognition model, it can monitor the state of the molten pool during welding in real time based on this molten pool profile information, identify abnormal situations in real time, so as to facilitate the user to take timely measures, reduce the defect rate, and improve the welding quality. In some embodiments, the anomaly detection includes, but is not limited to: extracting molten pool profile features from the molten pool profile information, and determining the corresponding welding detection results by analyzing the molten pool profile features; or, performing similarity matching between the molten pool profile and a preset standard profile, and determining the welding detection results based on the similarity matching results; or pre-training a corresponding anomaly recognition model (for example, using classifiers such as SVM and random forest to identify abnormal states), and using this anomaly recognition model to detect the molten pool profile information to determine the corresponding welding detection results.

[0053] In some embodiments, if the corresponding welding detection results are determined by analyzing the molten pool profile features, step S15 includes: step S151 (not shown), where device 1 determines the corresponding molten pool profile feature information based on the molten pool profile information, where the molten pool profile feature information matches the welding anomaly detection type to be detected; step S152 (not shown), where device 1 determines the welding detection result information corresponding to the welding anomaly detection type based on the molten pool profile feature information and the anomaly detection conditions matching the welding anomaly detection type.

[0054] In some embodiments, the welding anomaly detection type includes at least any one of the following: welding parameter anomaly detection; penetration state detection; weld deviation detection. Different welding anomaly detection types correspond to different molten pool profile feature information, and their welding detection results are also different.

[0055] In some embodiments, the welding anomaly detection type includes welding parameter anomaly detection, and the molten pool profile feature information includes molten pool profile deviation information; step S151 includes: determining molten pool profile geometric feature information based on the molten pool profile information; determining the molten pool profile deviation information based on the molten pool profile geometric feature information. For example, if the welding anomaly detection type includes welding parameter anomaly detection, then the molten pool profile feature information includes molten pool profile deviation information. The molten pool profile deviation information is used to characterize the deviation amplitude between the molten pool profile geometric feature information corresponding to the current molten pool profile information and the molten pool profile geometric feature information at the previous moment or the standard molten pool profile geometric feature information. This deviation information can be used to determine whether there are abnormal fluctuations in welding parameters (such as voltage, current, welding speed, etc.). In some embodiments, a double ellipsoid can be used (reference Figure 4The double-ellipsoid model schematic diagram shown is used to approximately simulate the welding molten pool. Based on this, combined with the molten pool contour information, the corresponding geometric feature information of the molten pool contour is determined. Among them, the geometric feature information of the molten pool contour can be represented by the feature vector X t and includes but is not limited to: the half-width b of the molten pool (half of the widest part of the molten pool), the length c1 of the front semi-axis (the semi-axis length of the front half-ellipse of the molten pool), the length c2 of the rear semi-axis (the semi-axis length of the rear half-ellipse of the molten pool), the tail support angle α (the included angle of the rear half-ellipse of the molten pool, that is, the included angle formed by the two points of the farthest point at the tail of the molten pool and the maximum width of the molten pool, and the normal range is 40° to 70°), the perimeter L, the area S, the fineness k of the molten pool (the ratio between the length of the molten pool and the maximum width of the molten pool (c1 + c2) / 2b), and the complexity θ (where θ = L^2 / 4πS, used to reflect the complexity of the molten pool contour). The molten pool contour deviation information e = |(X t - X t-1 ) / X t-1 | or e = |(X t - X0) / X0|. Among them, X t-1 represents the geometric feature information of the molten pool contour at the previous moment determined based on the molten pool contour information at the previous moment, and X0 represents the geometric feature information of the standard molten pool contour determined based on the preset standard molten pool contour information. The standard molten pool contour information can be obtained from the molten pool image information collected under the set ideal welding parameters (such as current, voltage, wire feeding speed, etc.). In some embodiments, if the welding anomaly detection type includes welding parameter anomaly detection, the anomaly detection conditions include: if the molten pool contour deviation information is greater than the preset deviation threshold, or the time ratio of the molten pool contour deviation information greater than the preset deviation threshold within a unit time exceeds the preset ratio threshold, it is determined that there is an anomaly; otherwise, there is no anomaly. If it is determined that there is an anomaly, the welding detection result information includes an anomaly warning. Further, the welding detection result information can also include the time information corresponding to the molten pool image information with the molten pool contour deviation information greater than the preset deviation threshold to facilitate the user to conduct anomaly troubleshooting. If no anomaly is detected, the welding detection result information includes normal welding.

[0056] In some embodiments, if the welding anomaly detection type includes penetration state detection, the molten pool profile feature information includes the molten pool profile area S. The anomaly detection conditions include: if the molten pool profile area is greater than the first area threshold, it is determined that there is an over-penetration defect; if the molten pool profile area is less than the second area threshold, it is determined that there is an under-penetration defect; otherwise, the welding is normal. Wherein, the second area threshold is less than the first area threshold. Accordingly, the welding detection result information includes over-penetration, under-penetration or normal. In some embodiments, if the welding anomaly detection type includes weld deviation detection, the molten pool profile feature information includes the molten pool profile area and the molten pool profile eccentricity angle (the molten pool profile eccentricity angle is the angle formed by the offset direction of the center of gravity (geometric center) of the molten pool relative to the welding path (or the theoretical weld center line)). The anomaly detection conditions include: if the molten pool profile area is within a preset area range and the molten pool profile eccentricity angle is greater than the preset angle threshold, it is determined that there is a weld deviation; if the molten pool profile area is within a preset area range and the molten pool profile eccentricity angle is less than or equal to the preset angle threshold, it is determined that the welding is normal; otherwise, further penetration state detection is required. Accordingly, the welding detection result information includes weld deviation, normal or detection suggestion.

[0057] In some embodiments, to smooth the molten pool profile, remove the interference of noise points, and improve the accuracy of the molten pool profile feature information, step S151 includes: Device 1 performs curve fitting on the molten pool profile information, and based on the molten pool profile information after curve fitting, determines the molten pool profile feature information. For example, Device 1 can perform curve fitting on the molten pool profile information based on a selected fitting model (such as polynomial fitting, ellipse fitting, or spline fitting, etc.), and calculate the fitting curve of the molten pool profile. This fitting curve is used as the molten pool profile information after curve fitting to calculate the corresponding molten pool profile feature information.

[0058] In some embodiments, the method further includes: step S16 (not shown), Device 1 determines the actual molten pool profile feature information based on the molten pool profile information; step S17 (not shown), Device 1 determines the corresponding welding parameter adjustment information based on the actual molten pool profile feature information and the molten pool profile feature specification value. In some embodiments, the actual molten pool profile feature information can be represented by a feature vector BW pred which includes, but is not limited to, the molten pool half-width, the front semi-axis length, the rear semi-axis length, the tail support angle, the perimeter, the area, the molten pool slenderness ratio, and the complexity. Here, the determination method of the actual molten pool profile feature information is the same as or similar to the method of determining the molten pool profile feature information in the foregoing step S151, so it will not be elaborated here and is included herein by reference. In some embodiments, the molten pool profile feature specification value is represented by a feature vector BW idealIt is represented by [specific representation]. It is the contour feature information of the ideal molten pool preset based on the current welding task and the corresponding historical welding experience data. Corresponding welding parameter adjustment information (such as welding speed adjustment information, welding current adjustment information, and / or welding voltage adjustment information, etc.) can be determined using corresponding optimization algorithms (such as gradient descent method, Bayesian optimization, etc.) to make the difference between the actual molten pool contour feature information and the molten pool contour feature specification value as small as possible. Thus, by real-time monitoring the molten pool contour information that can reflect the welding quality, the parameters can be quickly adjusted during the welding process to ensure that the welding quality meets the expected standard.

[0059] In some embodiments, the method further includes: step S18 (not shown), the device 1 determines the corresponding molten pool contour feature specification value based on the preset welding process specification. In some embodiments, the preset welding process specification includes welding parameters such as the ideal welding current, welding voltage, and welding speed. The preset welding process specification is pre-set by the user based on the current welding task and the corresponding historical welding experience data. Based on the welding process specification, the ideal welding conditions are simulated, and the ideal molten pool image information under this welding process specification is obtained. Based on the ideal molten pool image information, using the molten pool contour recognition model, the corresponding ideal molten pool contour information is obtained. Furthermore, based on this ideal molten pool contour information, the molten pool contour feature specification value is determined. Here, the method for determining the molten pool contour feature specification value is the same or similar to the methods of the foregoing steps S11, S12, and S151, so it will not be elaborated again and is included herein by reference.

[0060] In some embodiments, the step S17 includes: the device 1 uses a machine learning model to determine the first functional relationship between the molten pool contour feature information and the welding parameter information; based on the objective function and the first functional relationship, determine the second functional relationship regarding the welding parameter adjustment information, where the objective function corresponds to the difference between the actual molten pool contour feature information and the molten pool contour feature specification value; based on the second functional relationship, combined with the actual molten pool contour feature information and the molten pool contour feature specification value, determine the corresponding welding parameter adjustment information.

[0061] In some embodiments, the welding parameter information includes, but is not limited to, welding speed, welding current, and welding voltage. Based on a machine learning model (e.g., a multi-layer perceptron (MLP) model, a Transformer architecture, etc.), with the welding parameter information as the model input and the molten pool profile feature information as the model output, combined with corresponding training sample data, model training is carried out. The trained machine learning model is used to represent the first functional relationship. In some embodiments, the training sample data may be welding parameter sample data and corresponding molten pool profile feature data obtained from other similar welding productions before the start of the current welding. The training sample data may also be welding parameter sample data and corresponding molten pool profile feature data collected from the current welding production. Or the training sample data may also be a combination of the first two types of data. Using the training sample data for model training, the first functional relationship BW pred =σ(σ([S,C,V,T]*W1)*W2), where BW pred , S, C, V, T represent molten pool profile feature information, welding speed, welding current, welding voltage, and welding time respectively, and W1, W2 are model weight parameters.

[0062] In some embodiments, the goal of welding parameter adjustment is to make the difference between the actual molten pool profile feature information and the molten pool profile feature specification value as small as possible. Based on the actual molten pool profile feature information and the molten pool profile feature specification value, a corresponding objective function can be constructed. The objective function corresponds to the difference between the actual molten pool profile feature information and the molten pool profile feature specification value, and the goal of welding parameter adjustment is converted into minimizing the objective function. Based on the objective function, combined with the first functional relationship, using a corresponding optimization algorithm (e.g., gradient descent method, Bayesian optimization, etc.), the solution formula corresponding to the welding parameter adjustment information, that is, the second functional relationship, is determined to minimize the objective function. For example, by calculating the gradient of the objective function with respect to the welding parameter information at time k, the second functional relationship regarding the welding parameter adjustment information is obtained. Taking the objective function f=(BW pred -BW ideal ) 2 / 2 as an example, by calculating the gradients with respect to the welding parameter information S, C, V at time k, the second functional relationship regarding the welding parameter adjustment information (i.e., ΔS, ΔC, ΔV) is obtained:

[0063]

[0064] Substitute the actual molten pool profile feature information determined based on the collected molten pool profile information and the molten pool profile feature specification value into the above second functional relationship, and solve to obtain the welding parameter adjustment information that minimizes the objective function.

[0065] In some embodiments, the method further includes: step S19 (not shown), in which device 1 updates the welding parameters based on the welding parameter adjustment information. By monitoring the welding molten pool in real time, the welding parameters are adjusted in a timely manner to ensure that the welding quality meets the expected standards, realizing the adaptive adjustment of the welding parameters. In some embodiments, in order to ensure stable welding production, an adaptive adjustment period can be preset. For example, the adjustment period can be set to 0.5 seconds. Further, the time within the period can also be divided. For example, the first 0.4 seconds are used to obtain the molten pool image, use the molten pool contour recognition model to recognize the molten pool contour, calculate the molten pool contour feature information and the corresponding welding parameter adjustment information, and the last 0.1 seconds are used to update the welding parameters. For the molten pool images obtained within the period, the molten pool contour recognition model can be used to respectively recognize and calculate the molten pool contour feature information corresponding to each molten pool image, and the average value of these molten pool contour feature information is used as the actual molten pool contour feature information to solve and calculate the welding parameter adjustment information for this period to update the welding parameters.

[0066] Figure 5 FIG. shows a structural diagram of a device for recognizing molten pool contour information according to an embodiment of the present application. The device 1 includes a module 11 and a module 12. The module 11 obtains molten pool image information; the module 12 determines the corresponding molten pool contour information based on the molten pool image information by using a trained molten pool contour recognition model. Among them, the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to recognize the molten pool contour within the located area where the molten pool is located. Here, the Figure 5 The specific implementation manners corresponding to the shown module 11 and module 12 are the same as or similar to the specific embodiments of the foregoing steps S11 and S12 respectively, so they will not be described in detail again and are included herein by reference.

[0067] In some embodiments, the device 1 further includes a module 13 (not shown) and a module 14 (not shown). The module 13 obtains a molten pool contour tracking model through training using a target detection model based on a molten pool contour tracking sample data set, where the molten pool contour tracking sample data set includes first molten pool sample image information and molten pool area annotation information corresponding to the first molten pool sample image information; the module 14 obtains a molten pool contour detection model through training using a semantic segmentation model based on a molten pool contour detection sample data set in combination with the molten pool contour tracking model, where the molten pool contour detection model takes the output of the molten pool contour tracking model as input, and the molten pool contour detection sample data set includes second molten pool sample image information and molten pool contour annotation information corresponding to the second molten pool sample image information. Here, the specific implementation manners of the module 13 and the module 14 are the same as or similar to the specific implementation manners of the foregoing steps S13 and S14, so they will not be elaborated herein and are included herein by reference.

[0068] In some embodiments, the device 1 further includes a module 15 (not shown). The module 15 performs anomaly detection based on the molten pool contour information to obtain welding detection result information. Here, the specific implementation manner of the module 15 is the same as or similar to the specific implementation manner of the foregoing step S15, so it will not be elaborated herein and is included herein by reference.

[0069] In some embodiments, the module 15 includes a unit 151 (not shown) and a unit 152 (not shown). The unit 151 determines corresponding molten pool contour feature information based on the molten pool contour information, where the molten pool contour feature information matches the welding anomaly detection type to be detected; the unit 152 determines the welding detection result information corresponding to the welding anomaly detection type based on the molten pool contour feature information and an anomaly detection condition matching the welding anomaly detection type. Here, the specific implementation manners of the unit 151 and the unit 152 are respectively the same as or similar to the specific embodiments of the foregoing steps S151 and S152, so they will not be elaborated herein and are included herein by reference.

[0070] In some embodiments, the device 1 further includes a module 16 (not shown) and a module 17 (not shown). The module 16 determines actual molten pool contour feature information based on the molten pool contour information; the module 17 determines corresponding welding parameter adjustment information based on the actual molten pool contour feature information and a molten pool contour feature specification value. Here, the specific implementation manners of the module 16 and the module 17 are the same as or similar to the specific implementation manners of the foregoing steps S16 and S17, so they will not be elaborated herein and are included herein by reference.

[0071] In some embodiments, the device 1 further includes an eight module 18 (not shown). The eight module 18 determines corresponding molten pool profile feature specification values based on a preset welding process specification. Here, the specific implementation manner of the eight module 18 is the same as or similar to that of the foregoing step S18, so it will not be described in detail and is included herein by reference.

[0072] In some embodiments, the device 1 further includes a nine module 19 (not shown). The nine module 19 updates the welding parameters based on the welding parameter adjustment information. Here, the specific implementation manner of the nine module 19 is the same as or similar to that of the foregoing step S19, so it will not be described in detail and is included herein by reference.

[0073] Figure 6 An exemplary system that can be used to implement the various embodiments described in this application is shown;

[0074] As Figure 6 shown, in some embodiments, the system 300 can serve as any one of the devices in the various embodiments. In some embodiments, the system 300 may include one or more computer-readable media having instructions (e.g., system memory or NVM / storage device 320) and one or more processors (e.g., (one or more) processors 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement modules to perform the actions described in this application.

[0075] For one embodiment, the system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of the (one or more) processors 305 and / or any suitable device or component communicating with the system control module 310.

[0076] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0077] The system memory 315 can be used, for example, to load and store data and / or instructions for the system 300. For one embodiment, the system memory 315 may include any suitable volatile memory, e.g., suitable DRAM. In some embodiments, the system memory 315 may include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0078] For one embodiment, the system control module 310 may include one or more input / output (I / O) controllers to provide an interface to the NVM / storage device 320 and the (one or more) communication interfaces 325.

[0079] For example, the NVM / storage device 320 can be used to store data and / or instructions. The NVM / storage device 320 can include any suitable non-volatile memory (e.g., flash memory) and / or can include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more compact discs (CDs) drives, and / or one or more digital versatile discs (DVDs) drives).

[0080] The NVM / storage device 320 can include storage resources that are physically part of the device on which the system 300 is installed, or it can be accessed by the device without being part of the device. For example, the NVM / storage device 320 can be accessed via a network through the (one or more) communication interfaces 325.

[0081] The (one or more) communication interfaces 325 can provide an interface for the system 300 to communicate through one or more networks and / or with any other suitable device. The system 300 can wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols.

[0082] For one embodiment, at least one of the (one or more) processors 305 can be logically encapsulated with one or more controllers of the system control module 310 (e.g., the memory controller module 330). For one embodiment, at least one of the (one or more) processors 305 can be logically encapsulated with one or more controllers of the system control module 310 to form a system-in-package (SiP). For one embodiment, at least one of the (one or more) processors 305 can be logically integrated with one or more controllers of the system control module 310 on the same die. For one embodiment, at least one of the (one or more) processors 305 can be logically integrated with one or more controllers of the system control module 310 on the same die to form a system-on-chip (SoC).

[0083] In various embodiments, the system 300 can be, but is not limited to: a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, the system 300 can have more or fewer components and / or a different architecture. For example, in some embodiments, the system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0084] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium storing computer code, and when the computer code is executed, the method described in any of the previous items is executed.

[0085] The present application also provides a computer program product, and when the computer program product is executed by a computer device, the method described in any of the previous items is executed.

[0086] The present application also provides a computer device, which includes:

[0087] One or more processors;

[0088] A memory for storing one or more computer programs;

[0089] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method described in any of the previous items.

[0090] It should be noted that the present application can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the steps or functions described above. Similarly, the software program (including related data structures) of the present application can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and the like. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with the processor to execute each step or function.

[0091] In addition, a part of the present application can be applied as a computer program product, such as computer program instructions, and when executed by a computer, through the operation of the computer, the methods and / or technical solutions according to the present application can be invoked or provided. Those skilled in the art should understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0092] A communication medium includes a medium through which communication signals that include, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. The communication medium can include a guided transmission medium (such as cables and wires (e.g., optical fiber, coaxial, etc.)) and a wireless (unguided) medium that can propagate energy waves, such as sound, electromagnetic, RF, microwave, and infrared. The computer-readable instructions, data structures, program modules, or other data can be embodied as, for example, a modulated data signal in a wireless medium (such as a carrier wave or a similar mechanism that is part of what is embodied as spread spectrum technology). The term "modulated data signal" refers to a signal in which one or more of its characteristics are changed or set in a manner that encodes information in the signal. Modulation can be an analog, digital, or hybrid modulation technique.

[0093] By way of example, and not limitation, a computer-readable storage medium can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable storage medium includes, but is not limited to, volatile memory such as random access memory (RAM, DRAM, SRAM); and non-volatile memory such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, tapes, CDs, DVDs); or other media now known or later developed that are capable of storing computer-readable information / data for use by a computer system.

[0094] Here, an embodiment according to the present application includes an apparatus that includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein, when the computer program instructions are executed by the processor, the apparatus is triggered to operate based on the methods and / or technical solutions according to the foregoing multiple embodiments of the present application.

[0095] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed within the present application. Any reference signs in the claims should not be construed as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. A plurality of units or devices stated in the apparatus claims can also be implemented by one unit or device through software or hardware. First, second, etc. are used to denote names and do not denote any particular order.

Claims

1. A method for identifying molten pool profile information, wherein: The method comprises: Obtaining melt pool image information; Based on the molten pool image information, the corresponding molten pool contour information is determined using a trained molten pool contour recognition model, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour in the area where the located molten pool is located.

2. The method according to claim 1, wherein: The method further comprises: Based on a melt pool contour tracking sample data set, a melt pool contour tracking model is obtained by training a target detection model, wherein the melt pool contour tracking sample data set includes first melt pool sample image information and melt pool area annotation information corresponding to the first melt pool sample image information; Based on the melt pool contour detection sample dataset and combined with the melt pool contour tracking model, a semantic segmentation model is used to train a melt pool contour detection model, wherein the melt pool contour detection model takes the output of the melt pool contour tracking model as input, and the melt pool contour detection sample dataset includes second melt pool sample image information and melt pool contour annotation information corresponding to the second melt pool sample image information.

3. The method according to claim 1 or 2, wherein: The method further comprises: Based on the molten pool contour information, abnormality detection is performed to obtain welding detection result information.

4. The method according to claim 3, wherein: The performing of abnormality detection based on the molten pool contour information to obtain welding detection result information includes: Based on the molten pool contour information, determining corresponding molten pool contour feature information, wherein the molten pool contour feature information matches a welding anomaly detection type to be detected; Based on the molten pool contour feature information and an abnormality detection condition matching the welding abnormality detection type, welding detection result information corresponding to the welding abnormality detection type is determined.

5. The method according to claim 4, wherein: The welding anomaly detection type includes at least one of the following: Welding parameter anomaly detection; Melt penetration detection; Melt deviation detection.

6. The method according to any one of claims 1 to 5, wherein: The method further comprises: Based on the molten pool contour information, determining actual molten pool contour feature information; Based on the actual molten pool profile feature information and the molten pool profile feature standard value, corresponding welding parameter adjustment information is determined.

7. The method according to claim 6, wherein: The determining of corresponding welding parameter adjustment information based on the actual molten pool profile feature information and the molten pool profile feature standard value includes: Determine a first functional relationship between molten pool profile feature information and welding parameter information using a machine learning model; Determining a second functional relationship regarding welding parameter adjustment information based on an objective function and the first functional relationship, wherein the objective function corresponds to a difference between actual molten pool profile feature information and a molten pool profile feature specification value; Based on the second functional relationship, the corresponding welding parameter adjustment information is determined in combination with the actual molten pool profile feature information and the molten pool profile feature standard value.

8. A computer device for identifying molten pool profile information, comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images