A groove multi-category semantic segmentation method, system, storage medium and device

Abstract

The application discloses a kind of groove multi-category semantic segmentation method, system, storage medium and equipment, belong to image processing and intelligent welding technical field, method includes: S1.Acquisition groove image, and record welding information in groove image file name;S2.define the semantic region of multi-category in the groove image collected, and mark groove image, generate labeled dataset;S3.the welding information is integrated into the label corresponding to groove image in structured form;S4.build semantic segmentation neural network model of fusion welding information, the model is encoded by welding information Control output channel, shield the semantic category that does not comply with welding logic;S5.the dataset is used to train and verify the semantic segmentation neural network model, and the optimal semantic segmentation neural network model is obtained.The application introduces welding process information into semantic segmentation model, improves the recognition accuracy and robustness of model in complex welding scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing and intelligent welding technology, and in particular to a groove multi-class semantic segmentation method, system, storage medium and device. BACKGROUND

[0002] In the traditional molten pool monitoring field, passive visual imaging method is generally used to obtain the molten pool image in the welding process. This method uses the welding arc itself as the illumination light source, and the image quality is greatly affected by the arc light intensity fluctuation. When the arc is weak, only the molten pool area can be observed, and other areas in the groove are almost invisible due to the low brightness; when the arc is too strong, although the outline in the groove is relatively clear, the molten pool area often appears serious overexposure, resulting in the loss of molten pool boundary and topography information. Therefore, under the condition of passive vision, only the single information of the molten pool can be used for welding rail planning or welding parameter adjustment, and it is difficult to obtain more comprehensive visual information of the welding scene.

[0003] In order to solve the limitations of passive vision, in recent years, active illumination molten pool monitoring method has appeared. This method introduces external illumination in the camera system, realizes continuous light supplement for the welding area, and makes the molten pool and the internal structure of the groove can obtain stable and clear imaging at any time. Although the active illumination technology has been applied in some welding monitoring devices, the processing research on the images obtained by active illumination is still limited, especially the effective segmentation method for multi-class target areas in the groove is lacking. In the welding process, in addition to the molten pool, the information such as the groove boundary and the welding wire position in the groove is also important for welding quality evaluation and intelligent welding control. However, due to the lack of systematic definition and modeling of the semantic categories of these areas, the existing semantic segmentation methods usually only segment the single-class target of the molten pool, and it is difficult to meet the needs of multi-region identification in the active illumination environment.

[0004] In addition, the robustness of deep learning model in welding scene still has deficiencies. For example, in the non-welding state, the model may still misjudge the molten pool area, resulting in false judgment of the intelligent welding system. Therefore, the semantic segmentation method that simply depends on image information cannot fully adapt to the variable conditions of the welding process.

[0005] Therefore, there is an urgent need for a method based on a deep learning model that can define the multi-class semantic regions in the groove in detail and fuse the welding information, in order to meet the needs of high-performance welding monitoring and intelligent welding control. SUMMARY

[0006] The purpose of the present application is to overcome the technical problems existing in the prior art, and provide a groove multi-category semantic segmentation method, system, storage medium and equipment, which introduces welding process information (such as whether welding is being carried out, welding layer, etc.) into a semantic segmentation model, improves the understanding ability of the model to a real welding scene, and improves the accuracy and reliability of multi-category region identification.

[0007] The purpose of the present application is to overcome the technical problems existing in the prior art, and provide a groove multi-category semantic segmentation method, system, storage medium and equipment, which introduces welding process information (such as whether welding is being carried out, welding layer, etc.) into a semantic segmentation model, improves the understanding ability of the model to a real welding scene, and improves the accuracy and reliability of multi-category region identification.

[0008] In a first aspect, a groove multi-category semantic segmentation method is provided, comprising the following steps:

[0009] S1. Collecting a groove image and recording welding information in the groove image file name;

[0010] S2. Defining multi-category semantic regions in the collected groove image, and labeling the groove image to generate a labeled data set;

[0011] S3. Integrating the welding information into the corresponding label of the groove image in a structured form;

[0012] S4. Building a semantic segmentation neural network model that integrates welding information, which controls the output channel through welding information coding and shields semantic categories that do not conform to welding logic;

[0013] S5. Training and verifying the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

[0014] In some embodiments, the welding information includes at least one of welding state, welding layer and welding method.

[0015] In some embodiments, the multi-category semantic regions include background, groove side, groove surface, welding bead, molten pool, arc light, welding wire, welding in row, electrode, nozzle and tungsten electrode.

[0016] In some embodiments, the labeling of the groove image comprises:

[0017] The groove image is labeled using a polygon labeling method in Labelme software, the names and corresponding polygon vertices of each semantic region are recorded, and a json label file with the same name as the image file name is formed.

[0018] In some embodiments, the welding information is integrated into the corresponding label of the groove image in a structured form, comprising:

[0019] The welding information is represented as a json object using the key-value pair group method.

[0020] In some embodiments, the welding information is encoded as a tensor with a shape of [C, 1, 1], where C is the number of semantic categories, and the tensor takes values of 0 or 1, for controlling the activation state of the corresponding semantic category channel.

[0021] In some embodiments, the value of the welding information encoding is determined according to the welding information and welding professional knowledge rules, and the welding professional knowledge rules include:

[0022] No molten pool and arc light appears before welding;

[0023] Single-layer single-channel filling does not appear to be row welding;

[0024] The cover layer does not appear to be a beveled side;

[0025] The MIG / MAG welding method does not use a tungsten electrode.

[0026] In a second aspect, a bevel multi-category semantic segmentation system is provided, comprising:

[0027] An image acquisition module for acquiring a bevel image and recording welding information in the bevel image file name;

[0028] A data set generation module for defining multi-category semantic regions within the acquired bevel image and labeling the bevel image to generate a labeled data set;

[0029] A welding information fusion module for integrating the welding information into the corresponding label of the bevel image in a structured form;

[0030] A model construction module for constructing a semantic segmentation neural network model that integrates welding information, the model controls the output channel through welding information encoding and masks semantic categories that do not conform to welding logic;

[0031] A model training module for training and verifying the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

[0032] In a third aspect, a computer readable storage medium is provided, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the bevel multi-category semantic segmentation method of the first aspect.

[0033] In a fourth aspect, an electronic device is provided, comprising a memory and a processor, the memory stores computer instructions executable on the processor, and the processor executes the computer instructions to perform the bevel multi-category semantic segmentation method of the first aspect.

[0034] It should be further explained that the technical features corresponding to the above-mentioned various option embodiments can be combined or replaced with each other to form new technical solutions without conflict.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] 1. Improve the accuracy of semantic segmentation: By introducing welding information assistance, the semantic segmentation neural network model can dynamically adjust the recognition strategy according to the welding state, welding layer and other information, effectively avoid the misidentification problem under non-welding state, and significantly improve the accuracy of multi-class semantic segmentation.

[0037] 2. Enhance the robustness of the model: The welding information coding mechanism can shield the semantic categories that do not conform to the actual welding scene, enhance the adaptability of the semantic segmentation neural network model to complex and variable welding environment, and improve the stability of the system in practical application.

[0038] 3. Support multi-class collaborative recognition: For the first time, the multi-class semantic regions in the groove are defined and their collaborative segmentation is realized, providing more comprehensive visual information support for welding quality evaluation, process optimization and intelligent control.

[0039] 4. Suitable for various welding scenes: The method is compatible with different welding layers, welding methods and welding states, has strong universality and scalability, and is suitable for real-time monitoring and analysis of various welding processes such as MIG / MAG, TIG, etc. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 A flowchart of a groove multi-class semantic segmentation method of the present application;

[0041] Figure 2 A schematic diagram of the present application using an active illumination type molten pool monitoring camera to collect images;

[0042] Figure 3 A schematic diagram of the Labelme software labeling result of the present application;

[0043] Figure 4 A schematic diagram of the content of the Json label file corresponding to the Labelme software labeling result of the present application;

[0044] Figure 5 A schematic diagram of the content of the Json label file after integrating the welding information of the present application;

[0045] Figure 6 A schematic diagram of a general semantic segmentation model;

[0046] Figure 7 A schematic diagram of the semantic segmentation model proposed by the present application;

[0047] Figure 8A schematic diagram of a welding information coding table for the present application. DETAILED DESCRIPTION

[0048] The technical solutions of the present application will be described clearly and completely below with reference to the drawings. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0049] It should be noted that the defects of the above prior art solutions are the results obtained by the inventors after practice and careful study. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present application to the above problems should be the contributions made by the inventors to the present application in the process of invention and creation, and should not be understood as technical contents known to those skilled in the art.

[0050] To solve the technical problems pointed out in the background art, the embodiments provided by the present application are as follows:

[0051] In an exemplary embodiment, a multi-category semantic segmentation method for a groove is provided, as shown in Figure 1 The method comprises the following steps:

[0052] S1. Collecting groove images and recording welding information in the groove image file name;

[0053] S2. Defining multi-category semantic regions in the collected groove images and labeling the groove images to generate a labeled data set;

[0054] S3. Integrating the welding information into the label corresponding to the groove image in a structured form;

[0055] S4. Constructing a semantic segmentation neural network model that integrates welding information, which controls the output channels through welding information coding and screens semantic categories that do not conform to the welding logic;

[0056] S5. Training and verifying the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

[0057] Specifically, in step S1, the groove welding images before and during each layer of welding are collected using an active illumination type molten pool monitoring camera, as shown in Figure 2The welding information at the time of acquisition is added to the groove image file name, for example: "welding_ backing layer_ MIG / MAG welding_ 20251111X1_0.png". The welding information includes: welding state (before welding, during welding), welding layer (backing layer, filler layer, surface layer), welding method (MIG / MAG welding, TIG welding), etc.

[0058] In step S2, the data set is made. According to the statistics of a large number of various types of groove images, the target area in the groove scene can be summarized into the following 11 semantic categories: background, groove side, groove surface, weld, molten pool, arc light, welding wire, welding, electrode, nozzle, tungsten electrode. After the target area in the groove image is defined according to the above standard, a multi-class semantic area is obtained, and then based on the defined multi-class semantic area, the groove image collected is marked using the polygon marking method in the Labelme software. The marking result is as shown in Figure 3 After marking, a json label file with the same name as the groove image file name is generated, as shown in Figure 4 The json label file records the names and corresponding polygon vertices of the marked various class semantic areas. Finally, the marked data is divided into training set and validation set according to the ratio of T:V, wherein T and V are preset values, generally T=4 and V=1.

[0059] In step S3, the welding information is integrated into the data set label. First, read the groove image file name, and record the welding information in the groove image file name as a json object using the key-value pair group method, for example: {"welding state": "welding", "welding layer": "surface layer", "welding method": "MIG / MAG welding"}. Then read the json file corresponding to the image, and add the json object to the json array with the field name "weldInformation", for example: "weldInformation": [{"welding state": "welding", "welding layer": "surface layer", "welding method": "MIG / MAG welding"}]. Finally, add the "weldInformation" array to the json file for saving. The content in the saved json file is as shown in Figure 5

[0060] In step S4, the semantic segmentation neural network model to be trained is constructed. Generally, the structure of the semantic segmentation model is as shown in Figure 6 ​As shown, the encoder+decoder is composed, the input image is processed by the model, and an output feature map O1 with the same resolution (high x wide, i.e. H x W) as the input image is output. The number of channels C of the output feature map O1 is the number of semantic categories, and in the present application C = 11, each channel represents a semantic category. The semantic segmentation model constructed in the present application introduces welding information on the general model, and the value of a specific channel in the feature map output by the model can be controlled through the introduced welding information. According to the welding information, the feature channel of the semantic category that should not be output is shielded. The model constructed in the present application is as shown in Figure 7 As shown, first, a Relu activation layer is added to the output of the general model, so that the feature map O1 output by the general model has a value of 0 or more, and a feature map R1 is obtained. Then, a welding information coding tensor with shape [C, 1, 1] is added as input, where C is the number of semantic categories, and the value of the tensor can only be 0 or 1. The value is determined according to the welding information recorded in the input groove image file name and the welding professional knowledge rules, such as: the molten pool and arc light will not appear before welding; single-layer single-pass filling will not appear as a row weld; the groove side will not appear on the cover layer; MIG / MAG welding will not use a tungsten electrode. When these situations occur, the corresponding semantic category is set to 0. The specific welding information coding value is as shown in Figure 8 As shown, for example: the file name is "welding before_ cover layer_ gas shielded welding_ 20251111X1_0.png", according to the welding state in the file name, which is before welding, the value of the molten pool and arc light semantic class should be 0, and the value of the tensor is [1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1]. According to the file name, the welding layer is the cover layer, so the value of the groove side semantic class should be 0, and the value of the tensor is updated to [1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1]. According to the file name, the welding method is MIG / MAG, so the value of the tungsten electrode semantic class should be 0, and the value of the tensor is updated to [1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 0]. After determining the value of the welding information coding tensor, it is expanded to a feature map E1 with shape [C, H, W] through an expand operation in the model. Finally, the feature map R1 and the feature map E1 are multiplied element by element to obtain the final output O2 of the model. This method realizes the operation of setting the value of the semantic category feature channel that should not be output to 0 according to the welding information, thereby ensuring that the model does not misidentify.

[0061] The present application strongly associates the welding process information as metadata with the image, so that the model not only learns pixel features, but also perceives the welding context in which the image is located. For example, the model can explicitly know that the current image is a "cover layer" welding, so that when understanding the pixels, it will strengthen the identification of "welding bead", "molten pool", and weaken the irrelevant attention to "groove side". This information fusion mechanism enables the model to upgrade from "looking at pictures" to "understanding the welding scene", greatly improving the pertinence and accuracy of segmentation.

[0062] In another exemplary embodiment, based on the same inventive concept as the method embodiment, a groove multi-class semantic segmentation system is provided, comprising:

[0063] An image acquisition module is configured to acquire a groove image and record welding information in the groove image file name;

[0064] A data set generation module is configured to define multi-class semantic regions in the acquired groove image and label the groove image to generate a labeled data set;

[0065] A welding information fusion module is configured to integrate the welding information into the label corresponding to the groove image in a structured form;

[0066] A model construction module is configured to construct a semantic segmentation neural network model that integrates welding information, wherein the model controls the output channel through welding information coding and shields semantic categories that do not conform to welding logic;

[0067] A model training module is configured to train and verify the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

[0068] In another exemplary embodiment, based on the same inventive concept as the method embodiment, a computer readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements a groove multi-class semantic segmentation method provided by an embodiment of the present application. Based on this understanding, the technical solution of the present embodiment or the part that essentially contributes to the prior art or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method of the present application. The aforementioned storage medium includes U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, and various program code storage media.

[0069] In another exemplary embodiment, based on the same inventive concept as the method embodiment, an electronic device is provided, including a memory and a processor. The memory stores computer instructions that can be executed on the processor. When the processor executes the computer instructions, it performs a bevel multi-category semantic segmentation method provided in the embodiment of the present invention.

[0070] The processor may be a single-core or multi-core central processing unit or a specific integrated circuit, or one or more integrated circuits configured to implement the present invention.

[0071] The embodiments of the subject matter and functional operation described in this specification can be implemented in: tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing device or for controlling the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing device.

[0072] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0073] Suitable processors for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0074] It should be understood that each block of the flowchart of block diagram illustrations, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implement the functions specified in the flowchart and / or block diagram block or blocks.

[0075] The foregoing detailed description of the application has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the application to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. It is intended that the scope of the application be governed by the claims and their equivalents.

Claims

1. A method for bevel multi-class semantic segmentation, the method comprising: The method comprises the following steps: S1. Collecting a groove image and recording welding information in the groove image file name; S2. Defining multi-category semantic regions in the collected groove image and labeling the groove image to generate a labeled data set; S3. Integrating the welding information into the label corresponding to the groove image in a structured form; The welding information is integrated into the label corresponding to the groove image in a structured form, comprising: The welding information is represented as a json object in the form of key-value pair group; S4. Building a semantic segmentation neural network model integrating welding information, wherein the model controls the output channel through welding information coding and shields semantic categories that do not conform to welding logic; the model controls the output channel through welding information coding, comprising: The semantic segmentation neural network model adds a layer of Relu activation to the output of the general semantic segmentation model, so that the feature map O1 output by the general semantic segmentation model is all above 0, obtaining a feature map R1; The welding information coding is a tensor with a shape of [C, 1, 1], wherein C is the number of semantic categories, and the tensor takes a value of 0 or 1, which is determined according to the welding information recorded in the input groove image file name and welding professional knowledge rules, and is used to control the activation state of the corresponding semantic category channel; After determining the value of the welding information coding, an expand operation is performed on it in the model to expand it into a feature map E1 with a shape of [C, H, W]; Finally, the feature map R1 and the feature map E1 are subjected to Mul element-wise multiplication operation to obtain the final output O2 of the model; S5. Training and verifying the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

2. The multi-class semantic segmentation method of a groove according to claim 1, characterized in that, The welding information comprises at least one of welding state, welding layer and welding method.

3. The multi-class semantic segmentation method of a groove according to claim 1, characterized in that, The multi-category semantic regions comprise background, groove side, groove surface, weld, molten pool, arc light, welding wire, welding in row, electrode, nozzle and tungsten electrode.

4. The multi-class semantic segmentation method of a groove according to claim 1, characterized in that, The labeling of the groove image comprises: The groove image is labeled using the polygon labeling method in the Labelme software to record the names and corresponding polygon vertices of each semantic region, forming a json label file with the same name as the image file name.

5. The multi-class semantic segmentation method of a groove according to claim 1, characterized in that, The value of the welding information coding is determined according to the welding information and welding professional knowledge rules, and the welding professional knowledge rules comprise: The molten pool and arc light do not appear before welding; Single-layer single-pass filling does not appear in the row; The cover layer does not appear the groove side; The MIG / MAG welding method does not use the tungsten electrode.

6. A bevel multi-class semantic segmentation system, comprising: Comprise: An image acquisition module for acquiring a groove image and recording welding information in the groove image file name; A data set generation module for defining multi-category semantic regions in the collected groove image and labeling the groove image to generate a labeled data set; A welding information integration module for integrating the welding information into the label corresponding to the groove image in a structured form; The welding information is integrated into the label corresponding to the groove image in a structured form, comprising: The welding information is represented as a json object in the form of key-value pair group; A model construction module is configured to construct a semantic segmentation neural network model fused with welding information, the model controls the output channel through welding information coding and masks the semantic categories that do not conform to the welding logic; the model controls the output channel through welding information coding, which includes: The semantic segmentation neural network model adds a layer of Relu activation on the output of the general semantic segmentation model, so that the feature map O1 output by the general semantic segmentation model is above 0, and a feature map R1 is obtained; The welding information coding is a tensor with a shape of [C, 1, 1], where C is the number of semantic categories, and the tensor takes a value of 0 or 1, which is determined according to the welding information recorded in the input groove image file name and the welding professional knowledge rules, and is used to control the activation state of the corresponding semantic category channel; After determining the value of the welding information coding, an expand operation is performed on it in the model to expand it into a feature map E1 with a shape of [C, H, W]; Finally, the feature map R1 and the feature map E1 are subjected to a Mul element-by-element multiplication operation to obtain the final output O2 of the model; A model training module is configured to train and verify the semantic segmentation neural network model using the data set to obtain an optimal semantic segmentation neural network model.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: The computer program is executed by the processor to implement the groove multi-category semantic segmentation method of any one of claims 1-5.

8. An electronic device, comprising a memory and a processor, computer instructions stored on the memory and executable on the processor, characterized in that, The processor executes the computer instructions to implement the groove multi-category semantic segmentation method of any one of claims 1-5.

Images