Method and equipment for constructing molten pool contour recognition model, medium and program product

Abstract

The invention aims to provide a method and device for constructing a molten pool contour recognition model, a medium and a program product, the method comprises the steps that a sample data set corresponding to a molten pool image in the welding process is obtained, and the sample data set comprises a molten pool contour tracking sample data set and a molten pool contour detection sample data set; based on the sample data set, a molten pool contour recognition model is obtained through training, the molten pool contour recognition model comprises a molten pool contour tracking model and a molten pool contour detection model, the output of the molten pool contour tracking model serves as the input of the molten pool contour detection model, and the molten pool contour tracking model is used for positioning a molten pool area; the molten pool contour detection model is used for identifying a molten pool contour. According to the method, the molten pool area is rapidly positioned through the molten pool contour tracking model, and then the molten pool contour detection model is used for carrying out more detailed molten pool contour detection in the molten pool area, so that the anti-interference capability to noise is improved, and the image processing rate is improved while segmentation errors caused by backgrounds or other interference objects are avoided.

Description

Technical Field

[0001] This application relates to the field of welding, and particularly to a technology for constructing a molten pool contour recognition model. Background Art

[0002] With the development of image vision technology, more and more researchers have begun to pay attention to the changes in the molten pool during the welding process and attempt to establish a relationship between the change rules of the molten pool and the weld quality. Based on industry research, many contour features of the molten pool (such as weld width, area, trailing angle, etc.) can reflect the penetration and formation of the weld. In recent years, quality diagnosis and adaptive control of the welding process have been realized based on the processing of molten pool images and the extraction of contour features, and their applications in practice have become more and more extensive. However, whether it is quality diagnosis or weld tracking, it must be based on the real-time and accurate extraction of the contour features of the welding molten pool, which requires a complete and accurate method for molten pool contour tracking and recognition.

[0003] In recent years, the rapid development of deep learning has also brought new opportunities to molten pool recognition and detection. The introduction of deep learning models, especially various convolutional neural networks that perform well in the field of image processing (for example, Mask RCNN, Deeplabv3+, etc.), has significantly improved contour detection technology. However, in the actual application of molten pool contour detection, these methods are still limited by complex and changeable welding conditions. For example, in the face of interference in molten pool images such as arc light changes, relative camera viewing angle changes, complex backgrounds, and smoke interference, it may cause the model to be unable to accurately distinguish the molten pool from the background and result in misrecognition. Especially when the molten pool contour is relatively small, the recognition accuracy will decrease significantly. Summary of the Invention

[0004] An object of this application is to provide a method, device, medium, and program product for constructing a molten pool contour recognition model.

[0005] According to one aspect of this application, a method for constructing a molten pool contour recognition model is provided. The method includes:

[0006] Obtaining a sample data set corresponding to the molten pool image during the welding process, where the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set;

[0007] Based on the sample data set, training to obtain a molten pool contour recognition model, where the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour detection model takes the output of the molten pool contour tracking model as input, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to recognize the molten pool contour.

[0008] According to one aspect of the present application, there is provided a computer device for constructing a molten pool contour recognition model, including a memory, a processor, and a computer program stored on the memory, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0009] According to one aspect of the present application, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0010] According to one aspect of the present application, there is provided a computer program product including a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above methods.

[0011] According to one aspect of the present application, there is provided a device for constructing a molten pool contour recognition model, the device including:

[0012] a module for obtaining a sample data set corresponding to the molten pool image during the welding process, wherein the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set;

[0013] a second module for training and obtaining a molten pool contour recognition model based on the sample data set, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour detection model takes the output of the molten pool contour tracking model as input, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour.

[0014] Compared with the prior art, the present application obtains a sample data set corresponding to the molten pool image during the welding process, wherein the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set; based on the sample data set, a molten pool contour recognition model is trained and obtained, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour detection model takes the output of the molten pool contour tracking model as input, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour. The molten pool contour recognition model constructed by the present application can first quickly locate the area where the molten pool is located through the molten pool contour tracking model, and then use the molten pool contour detection model to further perform more detailed molten pool contour detection within the target area, improving the anti-interference ability against noise, avoiding segmentation errors caused by the background or other interfering objects, and further improving the image processing speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Other features, objects, and advantages of the present application will become more apparent from the following detailed description of non - limiting embodiments read in conjunction with the accompanying drawings:

[0016] Figure 1 A method flow chart for constructing a molten pool contour recognition model according to an embodiment of the present application is shown;

[0017] Figure 2A A schematic diagram of the annotation result of the molten pool area of a molten pool image frame according to an embodiment of the present application is shown;

[0018] Figure 2B A schematic diagram of the annotation result of the molten pool contour of a molten pool image frame according to an embodiment of the present application is shown;

[0019] Figure 3 A device structure diagram for constructing a molten pool contour recognition model according to an embodiment of the present application is shown;

[0020] Figure 4 An exemplary system that can be used to implement the various embodiments described in the present application is shown.

[0021] The same or similar reference numerals in the drawings represent the same or similar components. Detailed Description of the Specific Embodiment

[0022] The present application will be further described in detail below in conjunction with the accompanying drawings.

[0023] In a typical configuration of the present application, the terminal, the devices of the service network, and the trusted party all include one or more processors (e.g., a Central Processing Unit (CPU)), an input / output interface, a network interface, and a memory.

[0024] The memory may include non - permanent memory in a computer - readable medium, random access memory (RAM) and / or non - volatile memory in the form of, for example, read - only memory (ROM) or flash memory. The memory is an example of a computer - readable medium.

[0025] A computer-readable medium includes permanent and non-permanent, removable and non-removable media that can store information by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transitory medium that can be used to store information that can be accessed by a computing device.

[0026] The devices referred to in this application include, but are not limited to, user devices, network devices, or devices formed by integrating user devices and network devices through a network. The user devices include, but are not limited to, any mobile electronic product that can perform human-computer interaction with users (such as human-computer interaction through a touchpad), such as smart phones, tablets, etc. The mobile electronic products can adopt any operating system, such as Android operating system, iOS operating system, etc. Among them, the network devices include an electronic device that can automatically perform numerical calculations and information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc. The network devices include, but are not limited to, computers, network hosts, a single network server, a set of multiple network servers, or a cloud composed of multiple servers; here, the cloud is composed of a large number of computers or network servers based on cloud computing. Among them, cloud computing is a type of distributed computing, consisting of a virtual supercomputer formed by a group of loosely coupled computers. The network includes, but is not limited to, the Internet, wide area network, metropolitan area network, local area network, VPN network, wireless ad hoc network (Ad Hoc network), etc. Preferably, the device can also be a program running on the user device, network device, or a device formed by integrating user devices and network devices, network devices, touch terminals, or a device formed by integrating network devices and touch terminals through a network.

[0027] Of course, those skilled in the art should understand that the above devices are only examples. Other existing or future devices that may be applicable to this application should also be included within the protection scope of this application and are hereby incorporated by reference.

[0028] In the description of this application, the meaning of "a plurality" is two or more, unless otherwise specifically defined.

[0029] Figure 1The flowchart shows a method for constructing a molten pool contour recognition model according to an embodiment of the present application. The method includes step S11 and step S12. In step S11, device 1 obtains a sample data set corresponding to the molten pool image during the welding process. Among them, the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set. In step S12, device 1 trains and obtains a molten pool contour recognition model based on the sample data set. Among them, the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour detection model takes the output of the molten pool contour tracking model as input. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to recognize the molten pool contour.

[0030] In step S11, device 1 obtains a sample data set corresponding to the molten pool image during the welding process. Among them, the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set. In some embodiments, device 1 includes, but is not limited to, user devices and network devices with information processing or computing capabilities, such as tablet computers, computers, servers, etc. In some embodiments, the sample data set includes a molten pool contour tracking sample data set for training the molten pool contour tracking model and a molten pool contour detection sample data set for training the molten pool contour detection model. The molten pool contour tracking sample data set and the molten pool contour detection sample data set contain different annotation information. In some embodiments, the sample data set can be obtained by device 1 based on the molten pool video information transmitted from the welding site, or can be obtained by uploading the processed sample data set by other devices or users.

[0031] In some embodiments, step S11 includes: step S111 (not shown), device 1 obtains the molten pool video information during the welding process; step S112 (not shown), device 1 determines the sample data set based on the molten pool video information. Among them, the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set. For example, device 1 obtains the molten pool video information transmitted from a high-speed camera at the welding site. The molten pool video information records the dynamics of the molten pool during the actual welding process, including key stages such as the formation, flow, and cooling of the molten pool. Extract the molten pool image frames from the molten pool video information and annotate the molten pool image frames to generate the corresponding sample data set.

[0032] In some embodiments, step S112 includes: Device 1 determines corresponding molten pool image frames based on the molten pool video information; annotates the molten pool image frames to determine a corresponding sample data set, where the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set. The molten pool contour tracking sample data set includes the molten pool image frames and the molten pool region annotation information corresponding to the molten pool image frames. The molten pool contour detection sample data set includes the molten pool image frames and the molten pool contour annotation information corresponding to the molten pool image frames. In some embodiments, Device 1 extracts frames from the molten pool video information to obtain molten pool image frames for subsequent annotation. The molten pool image frames can be obtained by extracting frames at equal intervals (for example, extracting a number of molten pool image frames based on a set time / image frame number interval), or by extracting frames at random intervals (for example, first shuffling the order of all image frames in the molten pool video information and then extracting a number of molten pool image frames according to the set time / image frame number interval). Since the molten pool video information includes different stages such as the formation, flow, and cooling of the molten pool, to ensure that molten pool image frames in each stage are extracted, it is preferably to use the method of extracting frames at random intervals. In some embodiments, corresponding annotation tools (such as LabelMe (https: / / github.com / wkentaro / labelme), VIA (VGG Image Annotator, https: / / www.robots.ox.ac.uk / ~vgg / software / via / ), etc.) are used to annotate the molten pool image frames. In some embodiments, for any molten pool image frame, two types of annotation information, namely molten pool region annotation information and molten pool contour annotation information, are generated to be used for molten pool region detection and molten pool contour recognition respectively. Refer to Figure 2A , Figure 2B The schematic diagram of the annotation result of the molten pool image frame shown. In the molten pool image frame, the area where the molten pool is located is framed by a rectangular box. The area inside the box is the molten pool area, and the area outside the box is the background area (refer to Figure 2A ), and based on this, the molten pool region annotation information is obtained; in this molten pool image frame, the edge of the molten pool is depicted by a curve. The area inside the curve frame is the molten pool, and the area outside the curve frame is the background (refer to Figure 2B ), and based on this, the molten pool contour annotation information is obtained. In some embodiments, the molten pool region and the molten pool contour can also be respectively annotated for different molten pool image frames to obtain the corresponding molten pool contour tracking sample data set and molten pool contour detection sample data set.

[0033] In step S12, device 1 trains to obtain a molten pool contour recognition model based on the sample data set. Among them, the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour detection model takes the output of the molten pool contour tracking model as input. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to recognize the molten pool contour. For example, device 1 first trains to obtain a molten pool contour tracking model using the molten pool contour tracking sample data set, and the molten pool contour tracking model is used to locate the area where the molten pool is located. Then, based on this molten pool contour tracking model, the molten pool contour detection sample data set is used to train to obtain a molten pool contour detection model. The molten pool contour detection model can further segment within the molten pool area recognized by the molten pool contour tracking model to extract the precise contour of the molten pool. Thus, through the cooperation of the molten pool contour tracking model and the molten pool contour detection model, efficient and accurate molten pool contour recognition is achieved.

[0034] In some embodiments, step S12 includes: step S121 (not shown), device 1 trains to obtain a molten pool contour tracking model using a target detection model based on the molten pool contour tracking sample data set; step S122 (not shown), device 1 trains to obtain a molten pool contour detection model using a semantic segmentation model based on the molten pool contour detection sample data set in combination with the molten pool contour tracking model, where the molten pool contour detection model takes the output of the molten pool contour tracking model as input. For example, a suitable target detection model is selected (for example, all YOLO series models above YOLOv8), and a molten pool contour tracking model is iteratively trained based on the molten pool contour tracking sample data set. The molten pool image frames in the molten pool contour detection sample data set are input into the trained molten pool contour tracking model to determine the corresponding molten pool area information. This molten pool area information is input into the corresponding semantic segmentation model (for example, U-net model (a U-shaped structure semantic segmentation model composed of an encoder and a decoder), etc.), and the final molten pool contour detection model is iteratively trained in combination with the molten pool contour detection sample data set.

[0035] In some embodiments, the molten pool contour tracking sample data set and the molten pool contour detection sample data set can be divided according to a preset ratio respectively, so that the divided data can be used for model training, verification and testing respectively. In some embodiments, corresponding configuration files can be created based on the molten pool contour tracking sample data set and the molten pool contour detection sample data set respectively for the corresponding model training, so as to standardize and simplify the model training process. Taking the creation of a YAML format configuration file to describe the molten pool contour tracking sample data set as an example, its corresponding YAML file can be set as:

[0036] path:# Path

[0037] train: / path / to / data / train # Address of the training dataset for weld pool contour tracking

[0038] val: / path / to / data / val # Address of the validation dataset for weld pool contour tracking

[0039] test: / path / to / data / test # Address of the test dataset for weld pool contour tracking

[0040] names: # Class names

[0041] 0: background # Background

[0042] 1: weldpool # Weld pool

[0043] In some embodiments, a visualization tool (e.g., TensorBoard, etc.) can be used to monitor the training processes of the weld pool contour tracking model and the weld pool contour detection model, so as to help users understand the learning process of the models and debug the training of the models.

[0044] Here, those skilled in the art should understand that the above object detection model, semantic segmentation model, configuration file, and visualization tool are only examples. Other existing or future-emerging object detection models, semantic segmentation models, configuration files, and visualization tools that are applicable to this application should also be included within the protection scope of this application and are hereby incorporated herein by reference.

[0045] In some embodiments, step S121 includes: Device 1 performs iterative training using the object detection model based on the weld pool contour tracking sample dataset in combination with a first loss function to obtain a weld pool contour tracking model. For example, during the training process of the weld pool contour tracking model, based on the weld pool contour tracking sample dataset, the difference between the weld pool region information predicted by the weld pool contour tracking model and the true weld pool region annotation information is calculated in combination with the first loss function, and then the gradient of the loss function is calculated, thereby performing optimization iteration of the weld pool contour tracking model to obtain the final weld pool contour tracking model. In some embodiments, the first loss function includes a linear combination of the cross-entropy loss function and the intersection over union loss function. The cross-entropy loss function (Cross-Entropy Loss) is used to measure the difference between the probability distribution predicted by the model and the true distribution. The smaller the cross-entropy loss value, the closer the predicted probability of the model is to the true label, that is, the better the prediction effect of the model. The formula of the cross-entropy loss function can be expressed as:

[0046]

[0047] where C is the number of classes, y iis the true label, is the probability predicted by the model. The Intersection over Union Loss (IoU Loss) is used to measure the overlap degree between the predicted region and the true region. IoU (Intersection over Union) is defined as the ratio of the area of the intersection of the predicted region and the true region to the area of their union:

[0048]

[0049] where A is the predicted region and B is the true region. To make IoU applicable for optimization, the IoU loss is defined as: L IoU = 1 - IoU. The smaller the IoU loss value, the better the overlap between the region predicted by the model and the true region, that is, the better the prediction effect of the model. In some embodiments, a linear combination of the cross-entropy loss function and the Intersection over Union loss function can be performed based on actual requirements with a set weight ratio.

[0050] In some embodiments, the step S121 further includes: The device 1 evaluates and optimizes the molten pool profile tracking model based on a first evaluation metric. In some embodiments, the first evaluation metric includes at least any one of the following: mean average precision, precision, recall, F1-score. The mean average precision (mAP) is obtained by calculating the AP (average precision) for each category and then taking the average of all categories. AP refers to the average precision at different recall levels. For each category, a precision-recall curve can be plotted, and AP is the area under this curve. The precision is the ratio of the correctly predicted positive samples (TruePositive, TP) to all samples predicted as positive samples, Precision = TP / (TP + FP), where TP is the true positive, that is, the number of samples correctly predicted as the positive class by the model, and FP is the false positive, that is, the number of samples incorrectly predicted as the positive class by the model. The recall is the ratio of the correctly predicted positive samples to all actual positive samples, Recall = TP / (TP + FN), where FN is the false negative, that is, the number of positive class samples incorrectly predicted as the negative class by the model. The F1-score is the harmonic mean of precision and recall, F1 = 2 × (Precision × Recall) / (Precision + Recall). During the training process of the molten pool profile tracking model, the above first evaluation metric is used to quantify the performance of the model. The hyperparameters of the molten pool profile tracking model (such as learning rate, optimizer, regularization parameter, etc.) can be adjusted by the method of controlling variables in combination with the calculation results of the first evaluation metric, and techniques such as grid search and random search can be used to systematically explore the hyperparameter space to find the optimal combination of hyperparameters, so as to obtain the optimal molten pool profile tracking model.

[0051] In some embodiments, the step S122 includes: step S1221 (not shown), the device 1 determines corresponding molten pool area information based on the molten pool image frame by using the molten pool contour tracking model; step S1222 (not shown), the device 1 combines the second loss function based on the molten pool area information and the molten pool contour detection sample data set, and uses a semantic segmentation model to train and obtain a molten pool contour detection model. For example, the molten pool image frame in the molten pool contour detection sample data set is input into the molten pool contour tracking model to determine the corresponding molten pool area information. During the training process of the molten pool contour detection model, the molten pool area information is used as the model input to predict the molten pool contour information. Based on the second loss function, the difference between the predicted molten pool contour information and the true molten pool contour annotation information is calculated, and then the gradient of the loss function is calculated, so as to perform the optimization iteration of the molten pool contour detection model to obtain the final molten pool contour tracking model. In some embodiments, the second loss function includes a linear combination of the Dice similarity loss function and the cross-entropy loss function. The Dice similarity loss function (Dice Loss) is based on the Dice coefficient and is used to measure the similarity between the predicted region and the true region. The Dice loss is sensitive to small targets or class imbalance situations and can well optimize the segmentation effect of small-area targets. The Dice coefficient is usually defined as: Dice = 2×|A∩B| / (|A| + |B|) = 2×TP / (2×TP + FP + FN). The Dice similarity loss function is defined as: Dice Loss = 1 - Dice. The smaller the Dice Loss value, the higher the similarity between the model prediction region and the true region, that is, the better the prediction effect of the model. The cross-entropy loss function is the same as that in the aforementioned step S121, so it will not be elaborated here and is included herein by reference. In some embodiments, based on actual requirements, a linear combination of the Dice similarity loss function and the cross-entropy loss function can be performed with a set weight ratio.

[0052] In some embodiments, the step S1222 includes: the device 1 combines the second loss function based on the molten pool area information and the molten pool contour detection sample data set, and uses a semantic segmentation model to train and obtain a molten pool contour detection reference model; performs data cropping optimization on the molten pool contour detection sample data set to obtain a corresponding cropped and optimized sample data set; uses the cropped and optimized sample data set for model training to update the molten pool contour detection reference model; performs multiple data augmentation optimizations on the cropped and optimized sample data set to obtain corresponding multiple augmented and optimized sample data sets; based on the updated molten pool contour detection reference model, uses the multiple augmented and optimized sample data sets to train and obtain a molten pool contour detection optimized model; performs model parameter optimization based on the molten pool contour detection optimized model to obtain a molten pool contour detection model.

[0053] For example, first, according to the molten pool contour detection sample data set, based on the hyperparameters (such as learning rate, optimizer, regularization parameter, etc.) of a pre-set semantic segmentation model, combined with a second loss function, train to obtain a molten pool contour detection benchmark model. Then, crop the molten pool image frames in the molten pool contour detection sample data set used for training, remove the redundant background interference in the image, highlight the position of the molten pool, and obtain a cropped and optimized sample data set. Keeping other parameters unchanged during the training process, use the cropped and optimized sample data set for model training to obtain a new molten pool contour detection model. Based on a second evaluation metric (such as IoU and Dice weighted and combined according to a preset weight), compare the new molten pool contour detection model with the previously trained molten pool contour detection benchmark model, and select the model with better performance as the new molten pool contour detection benchmark model. In some embodiments, the multiple data augmentation optimizations for the cropped and optimized sample data set are to process the cropped and optimized sample data set based on a data augmentation method to obtain a corresponding augmented and optimized sample data set, and then add a new data augmentation method on the basis of the augmented and optimized sample data set obtained from the previous data augmentation optimization to obtain a corresponding augmented and optimized sample data set. By gradually adding new data augmentation methods, multiple augmented and optimized sample data sets are obtained. The data augmentation methods include but are not limited to rotation (for example, randomly rotating the image, increasing the samples of the molten pool at different angles, and helping the model learn the features of the molten pool in various directions), scaling (for example, randomly scaling the image, simulating the features of the molten pool at different distances, and enhancing the adaptability of the model to molten pools of different sizes), flipping (for example, horizontally and vertically flipping the image, increasing sample diversity, and helping the model learn symmetric features), cropping (for example, randomly cropping image regions, ensuring that the model can focus on different parts of the molten pool, and improving the detection ability for small targets), brightness and contrast adjustment (for example, randomly adjusting the brightness and contrast of the image, enhancing the robustness of the model under different lighting conditions, especially the lighting changes that may occur during the welding process), noise addition (for example, adding Gaussian noise or other types of noise to the image, helping the model learn its performance in an interference environment, and improving the anti-interference ability), random occlusion (for example, simulating the situation where part of the molten pool is occluded, and enhancing the model's ability to process partially visible targets). On the basis of the aforementioned molten pool contour detection benchmark model, train respectively based on each augmented and optimized sample data set to obtain candidate molten pool contour detection optimization models corresponding to each augmented and optimized sample data set. Based on the second evaluation metric, select the model with better performance from each candidate molten pool contour detection optimization model and the aforementioned molten pool contour detection benchmark model as the molten pool contour detection optimization model. Here, using various data augmentation methods effectively expands the training data set and improves the detection ability and adaptability of the model for the molten pool contour.In some embodiments, techniques such as grid search and random search can also be used to continue optimizing the model parameters, adjusting the hyperparameters of the model, finding the optimal combination of hyperparameters, and thus obtaining the optimal molten pool contour detection model.

[0054] In some embodiments, to improve the model training efficiency and performance, before obtaining a plurality of enhanced and optimized sample data sets corresponding to the cropped and optimized sample data set through multiple data augmentation optimizations and training the molten pool contour detection optimized model using the plurality of enhanced and optimized sample data sets based on the updated molten pool contour detection benchmark model, the step S1222 further includes: normalizing the cropped and optimized sample data set. By normalizing, pixel values of different scales are converted to the same range (such as [0, 1] or [-1, 1], etc.), which helps to accelerate the convergence speed of the model; avoids the influence of large-range pixel values on the convergence of the loss function and effectively avoids gradient explosion; effectively reduces the deviation of the model due to the large difference in the pixel distribution of the input data; enhances the data augmentation effect and helps to generate more representative samples. The ways of the normalization process include but are not limited to min-max normalization, Z-score normalization, and max absolute value normalization.

[0055] In some embodiments, the method further includes: step S13 (not shown), the device 1 obtains target molten pool image information; based on the target molten pool image information, using the molten pool contour recognition model, determines the corresponding target molten pool contour information. For example, after the molten pool contour recognition model is trained, the device 1 deployed with the molten pool contour recognition model can obtain in real time the target molten pool image information captured by a high-speed camera during the welding process from the welding site for real-time recognition of the molten pool contour. Based on the target molten pool image information captured in real time on site, the target molten pool region information in the target molten pool image information is first input into the molten pool contour tracking model. Then the target molten pool region information is input into the molten pool contour detection model to obtain the target molten pool contour information. Here, first, the irrelevant background information in the image is filtered through the molten pool contour tracking model to quickly locate the specific area where the molten pool is located; then the molten pool contour detection model is used to perform detailed segmentation and identify the molten pool contour within the molten pool area, which can improve the model segmentation accuracy, reduce the consumption of model computing resources, and lower the computing power cost.

[0056] Figure 3The structural diagram of a device for constructing a molten pool contour recognition model according to an embodiment of the present application is shown. The device 1 includes a module 11 and a module 12. The module 11 obtains a sample data set corresponding to the molten pool image during the welding process. Among them, the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set; the module 12 trains and obtains a molten pool contour recognition model based on the sample data set. Among them, the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model. The molten pool contour detection model takes the output of the molten pool contour tracking model as input. The molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to recognize the molten pool contour. Here, the Figure 3 The specific implementation manners corresponding to the shown module 11 and module 12 are the same as or similar to the specific embodiments of the foregoing step S11 and step S12 respectively, so they will not be described in detail and are included herein by reference.

[0057] In some embodiments, the module 11 includes a unit 111 (not shown) and a unit 112 (not shown). The unit 111 obtains the molten pool video information during the welding process; the unit 112 determines the sample data set based on the molten pool video information. Among them, the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set. Here, the specific implementation manners of the unit 111 and the unit 112 are the same as or similar to the specific embodiments of the foregoing step S111 and step S112 respectively, so they will not be described in detail and are included herein by reference.

[0058] In some embodiments, the module 12 includes a unit 121 (not shown) and a unit 122 (not shown). The unit 121 trains and obtains a molten pool contour tracking model based on the molten pool contour tracking sample data set by using an object detection model; the unit 122 trains and obtains a molten pool contour detection model based on the molten pool contour detection sample data set in combination with the molten pool contour tracking model by using a semantic segmentation model. Among them, the molten pool contour detection model takes the output of the molten pool contour tracking model as input. Here, the specific implementation manners of the unit 121 and the unit 122 are the same as or similar to the specific embodiments of the foregoing step S121 and step S122 respectively, so they will not be described in detail and are included herein by reference.

[0059] In some embodiments, the 122 unit 122 includes a 122 first sub-unit 1221 (not shown) and a 122 second sub-unit 1222 (not shown). The 122 first sub-unit 1221 determines corresponding molten pool area information based on the molten pool image frame and using the molten pool contour tracking model; the 122 second sub-unit 1222 trains and obtains a molten pool contour detection model by using a semantic segmentation model based on the molten pool area information, the molten pool contour detection sample data set, and in combination with a second loss function. Herein, the specific implementation manners of the 122 first sub-unit 1221 and the 122 second sub-unit 1222 are the same as or similar to the specific embodiments of the foregoing step S1221 and step S1222 respectively, and thus will not be described in detail again and are included herein by reference.

[0060] In some embodiments, the device 1 further includes a 13 module 13 (not shown). The 13 module 13 obtains target molten pool image information; and determines corresponding target molten pool contour information based on the target molten pool image information and using the molten pool contour recognition model. Herein, the specific implementation manner of the 13 module 13 is the same as or similar to the specific implementation manner of the foregoing step S13, and thus will not be described in detail again and are included herein by reference.

[0061] Figure 4 An exemplary system that can be used to implement the various embodiments described in the present application is shown; as Figure 4 shown, in some embodiments, the system 300 can serve as any one of the devices in the various embodiments. In some embodiments, the system 300 may include one or more computer-readable media having instructions (e.g., a system memory or the NVM / storage device 320) and one or more processors (e.g., (one or more) processors 305) coupled to the one or more computer-readable media and configured to execute the instructions to implement modules so as to perform the actions described in the present application.

[0062] For one embodiment, the system control module 310 may include any suitable interface controller to provide any suitable interface to at least one of the (one or more) processors 305 and / or any suitable device or component communicating with the system control module 310.

[0063] The system control module 310 may include a memory controller module 330 to provide an interface to the system memory 315. The memory controller module 330 may be a hardware module, a software module, and / or a firmware module.

[0064] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. For one embodiment, system memory 315 can include any suitable volatile memory, such as, for example, suitable DRAM. In some embodiments, system memory 315 can include double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

[0065] For one embodiment, system control module 310 can include one or more input / output (I / O) controllers to interface with NVM / storage device 320 and (one or more) communication interfaces 325.

[0066] For example, NVM / storage device 320 can be used to store data and / or instructions. NVM / storage device 320 can include any suitable non-volatile memory (such as, for example, flash memory) and / or can include any suitable (one or more) non-volatile storage devices (such as, for example, one or more hard disk drives (HDDs), one or more compact discs (CDs) drives, and / or one or more digital versatile discs (DVDs) drives).

[0067] NVM / storage device 320 can include storage resources that are physically part of the device on which system 300 is mounted, or it can be accessed by the device without being part of the device. For example, NVM / storage device 320 can be accessed via (one or more) communication interfaces 325 over a network.

[0068] (One or more) communication interfaces 325 can provide an interface for system 300 to communicate over one or more networks and / or with any other suitable device. System 300 can wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols.

[0069] For one embodiment, at least one of (one or more) processors 305 can be logically encapsulated with one or more controllers of system control module 310 (such as, for example, memory controller module 330). For one embodiment, at least one of (one or more) processors 305 can be logically encapsulated with one or more controllers of system control module 310 to form a system in package (SiP). For one embodiment, at least one of (one or more) processors 305 can be logically integrated with one or more controllers of system control module 310 on the same die. For one embodiment, at least one of (one or more) processors 305 can be logically integrated with one or more controllers of system control module 310 on the same die to form a system on chip (SoC).

[0070] In various embodiments, system 300 can be, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). In various embodiments, system 300 can have more or fewer components and / or a different architecture. For example, in some embodiments, system 300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an application specific integrated circuit (ASIC), and a speaker.

[0071] In addition to the methods and devices described in the above embodiments, the present application also provides a computer-readable storage medium storing computer code, and when the computer code is executed, the method as described in any one of the preceding items is executed.

[0072] The present application also provides a computer program product, and when the computer program product is executed by a computer device, the method as described in any one of the preceding items is executed.

[0073] The present application also provides a computer device, which includes:

[0074] One or more processors;

[0075] A memory for storing one or more computer programs;

[0076] When the one or more computer programs are executed by the one or more processors, the one or more processors are caused to implement the method as described in any one of the preceding items.

[0077] It should be noted that the present application can be implemented in software and / or a combination of software and hardware. For example, it can be implemented using an application specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In one embodiment, the software program of the present application can be executed by a processor to implement the above-described steps or functions. Similarly, the software program (including related data structures) of the present application can be stored in a computer-readable recording medium, such as a RAM memory, a magnetic or optical drive, or a floppy disk and the like. Additionally, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to execute each step or function.

[0078] In addition, a part of this application can be applied as a computer program product, such as computer program instructions, which when executed by a computer, can call or provide the methods and / or technical solutions according to this application through the operations of the computer. Those skilled in the art should understand that the forms of existence of computer program instructions in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways for a computer to execute computer program instructions include, but are not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0079] The communication medium includes a medium through which communication signals containing, for example, computer-readable instructions, data structures, program modules, or other data are transmitted from one system to another. The communication medium can include guided transmission media (such as cables and wires (e.g., optical fibers, coaxial cables, etc.)) and wireless (unguided) media that can propagate energy waves, such as sound, electromagnetic, RF, microwave, and infrared. Computer-readable instructions, data structures, program modules, or other data can be embodied as, for example, a modulated data signal in a wireless medium (such as a carrier wave or a similar mechanism that is part of spread spectrum technology). The term "modulated data signal" refers to a signal whose one or more characteristics are changed or set in a way that encodes information in the signal. Modulation can be analog, digital, or a hybrid modulation technique.

[0080] By way of example and not limitation, the computer-readable storage medium can include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable storage medium includes, but is not limited to, volatile memory such as random access memory (RAM, DRAM, SRAM); and non-volatile memory such as flash memory, various read-only memories (ROM, PROM, EPROM, EEPROM), magnetic and ferromagnetic / ferroelectric memories (MRAM, FeRAM); and magnetic and optical storage devices (hard disks, tapes, CDs, DVDs); or other media known now or developed in the future that can store computer-readable information / data for use by a computer system.

[0081] Here, an embodiment according to this application includes a device that includes a memory for storing computer program instructions and a processor for executing the program instructions. Wherein, when the computer program instructions are executed by the processor, the device is triggered to run based on the methods and / or technical solutions according to the foregoing multiple embodiments of this application.

[0082] For those skilled in the art, it is obvious that the present application is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of the present application, the present application can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting. The scope of the present application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be embraced within the present application. Any reference signs in the claims should not be construed as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. The multiple elements or devices stated in the apparatus claims can also be implemented by one element or device through software or hardware. The words such as "first" and "second" are used to denote names and do not denote any particular order.

Claims

1. A method for constructing a molten pool contour recognition model, wherein: The method comprises: Acquire a sample data set corresponding to a molten pool image during welding, wherein the sample data set includes a molten pool contour tracking sample data set and a molten pool contour detection sample data set; Based on the sample data set, a molten pool contour recognition model is trained, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour detection model takes the output of the molten pool contour tracking model as input, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour.

2. The method according to claim 1, wherein: Based on the sample data set, a molten pool contour recognition model is trained and obtained, wherein the molten pool contour recognition model includes a molten pool contour tracking model and a molten pool contour detection model, the molten pool contour detection model uses the output of the molten pool contour tracking model as input, the molten pool contour tracking model is used to locate the area where the molten pool is located, and the molten pool contour detection model is used to identify the molten pool contour, including: Based on the molten pool contour tracking sample data set, a molten pool contour tracking model is obtained by training a target detection model; Based on the melt pool contour detection sample data set and in combination with the melt pool contour tracking model, a semantic segmentation model is used to train a melt pool contour detection model, wherein the melt pool contour detection model uses the output of the melt pool contour tracking model as input.

3. The method according to claim 2, wherein: The method of acquiring a molten pool contour tracking model by training a target detection model based on the molten pool contour tracking sample data set includes: Based on the molten pool contour tracking sample data set and in combination with the first loss function, the target detection model is used for iterative training to obtain a molten pool contour tracking model.

4. The method according to claim 3, wherein: The method of acquiring a molten pool contour tracking model by training a target detection model based on the molten pool contour tracking sample data set also includes: Based on the first evaluation index, the molten pool contour tracking model is evaluated and optimized.

5. The method according to claim 2, wherein: The melt pool contour detection sample data set is combined with the melt pool contour tracking model, and a melt pool contour detection model is obtained by training a semantic segmentation model, wherein the melt pool contour detection model uses the output of the melt pool contour tracking model as input and includes: Based on the molten pool image frame, using the molten pool contour tracking model, determining corresponding molten pool area information; Based on the molten pool area information and the molten pool contour detection sample data set, combined with the second loss function, a semantic segmentation model is used to train a molten pool contour detection model.

6. The method according to claim 5, wherein: The method of acquiring a molten pool contour detection model by training a semantic segmentation model based on the molten pool area information and the molten pool contour detection sample data set in combination with a second loss function includes: Based on the molten pool area information and the molten pool contour detection sample data set, combined with the second loss function, a semantic segmentation model is used to train a molten pool contour detection benchmark model; Performing data trimming and optimization on the sample data set for detecting the molten pool contour, and obtaining a corresponding trimmed and optimized sample data set; performing model training using the trimmed and optimized sample data set, and updating the benchmark model for detecting the molten pool contour; Performing multiple data enhancement optimization on the cropped optimized sample data set to obtain a corresponding plurality of enhanced optimized sample data sets; based on the updated molten pool contour detection benchmark model, using the plurality of enhanced optimized sample data sets to train and obtain a molten pool contour detection optimization model; Model parameters are optimized based on the molten pool contour detection optimization model to obtain a molten pool contour detection model.

7. The method according to any one of claims 1 to 6, wherein: The method further comprises: Obtain target molten pool image information; Based on the target molten pool image information, the corresponding target molten pool contour information is determined using the molten pool contour recognition model.

8. A computer device for constructing a molten pool contour recognition model, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images