A wire fusion mark metallographic image recognition method based on semantic segmentation

Abstract

This invention provides a metallographic image recognition method for wire weld marks based on semantic segmentation, comprising the following steps: preprocessing the metallographic image of the wire weld mark; performing semantic segmentation to obtain the image of the molten zone in the feature set of the weld mark; sequentially performing binarization, morphological etching, and median filtering to extract the hole region and calculate the geometric features of the hole region; selecting the largest rectangular region in the molten zone image after removing the hole region as the grain region, and calculating the LBP texture features and GLCM texture features of the grain region; inputting the geometric features, LBP texture features, and GLCM texture into a classification model, and the classification model outputs the weld mark category. This invention can recognize the metallographic image of the wire weld mark obtained directly from the photograph of the metallographic sample, achieving the classification of four types of weld mark metallographic images, without additional human intervention and expanding the range of weld marks that can be recognized, thus improving the accuracy and efficiency of recognition.

Description

Technical Field

[0001] This invention belongs to the field of metallographic image recognition technology for wire weld marks, and specifically relates to a method for metallographic image recognition of wire weld marks based on semantic segmentation. Background Technology

[0002] Statistics show that electrical fires account for 28% of all fires annually, posing a significant threat to people's property and lives. Analysis of the causes of electrical fires reveals that a large proportion are due to wiring faults. When a short circuit occurs, the equipment temperature rises rapidly within a short period, reaching the ignition point of a flammable material and thus causing a fire. Firefighters observe and analyze the melting marks on the wires collected at the fire scene to identify the type of melting marks and thus infer the cause of the accident, providing strong evidence for determining the cause of the fire and liability.

[0003] Existing studies on metallographic images of wire melt marks have all undergone local processing and magnification, and manually removed interference. The resulting metallographic images only contain clear and complete grain structures in the melted area. The classification and recognition task is completed by extracting features through simple image preprocessing, and only two types of melt marks, namely primary short-circuit melt marks and secondary short-circuit melt marks, can be identified. Summary of the Invention

[0004] This invention addresses the technical problems existing in the prior art by providing a metallographic image recognition method for wire melt marks based on semantic segmentation. It can directly capture images of wire melt marks from samples collected at fire scenes, and then perform metallographic processing on the resulting metallographic images for recognition. This method achieves the classification of four types of metallographic images of melt marks, requires no additional human intervention, expands the range of melt marks that can be recognized, and improves the accuracy and efficiency of recognition.

[0005] The technical solution adopted in this invention is: a metallographic image recognition method for wire melt marks based on semantic segmentation, comprising the following steps:

[0006] Step 1: Preprocess the metallographic image of the wire melt mark;

[0007] Step 2: Perform semantic segmentation on the preprocessed metallographic image of the wire melt mark to obtain the image of the melting zone in the melt mark feature set;

[0008] Step 3: Perform binarization, morphological erosion, and median filtering on the image of the molten area in sequence to extract the hole region and calculate the geometric features of the hole region;

[0009] Step 4: Select the largest rectangular region in the melt zone image after removing the hole region as the grain region, and calculate the LBP texture features and GLCM texture features of the grain region;

[0010] Step 5: Input the geometric features of the hole region, the LBP texture features and GLCM texture features of the grain region into the classification model, and the classification model outputs the melt mark category.

[0011] Furthermore, in step 1, preprocessing: the metallographic image of the wire melt mark is normalized to 512*512 pixels, and black pixels are used to ensure that the aspect ratio of the image remains unchanged.

[0012] Furthermore, in step 2, the TransUnet neural network is used to perform semantic segmentation on the preprocessed metallographic image of the wire melt mark.

[0013] Furthermore, in step 3, the OTSU algorithm is used to binarize the image of the melting area to obtain a binary image. The difference between the semantic segmentation image mask in step 2 and the binary image is obtained. The difference image is iterated multiple times using circular structured pixel blocks in a morphological erosion manner to remove the connected regions between holes, and the holes are separated and extracted. Then, median filtering is used to remove discrete points in the image and extract the hole region.

[0014] Furthermore, in step 3, the geometric characteristics of the hole region include average area, average perimeter, roundness, number, and percentage of total area.

[0015] Furthermore, in step 4, the LBP algorithm is used to encode the grain region to obtain the LBP texture image, and the number of each pixel in the LBP texture image is counted using a histogram as the LBP texture feature.

[0016] Furthermore, in step 4, the GLCM texture features include energy, contrast, correlation, entropy, and homogeneity along four directions: 0°, 45°, 90°, and 135°.

[0017] Furthermore, in step 5, the geometric features of the hole region, the LBP texture features and GLCM texture features of the grain region are normalized based on the area of ​​the melt zone image, and then input into the classification model.

[0018] Furthermore, in step 5, the classification model is the XGBoost classifier.

[0019] Furthermore, in step 5, the melt mark categories are primary short-circuit melt mark, secondary short-circuit melt mark, equiaxed crystal melt mark, and eutectic melt mark.

[0020] Compared with existing technologies, the advantages of this invention are as follows: The metallographic images of conductor melt marks addressed in this invention are directly captured from fire scenes after metallographic processing, making direct classification, identification, and feature extraction impossible using existing methods. This invention uses neural networks to perform semantic segmentation on the metallographic images of conductor melt marks, obtaining images containing only the molten zone. It then combines multiple features of pores and grains to intelligently identify the melt mark categories, achieving fully automated identification of conductor melt mark metallographic images. Compared to existing methods, this invention is simpler and more direct, requires no additional manual processing, and offers wider, more accurate, and more efficient identification of different types. Attached Figure Description

[0021] Figure 1 This is a flowchart of an embodiment of the present invention;

[0022] Figure 2 This is a preprocessed metallographic image of the wire melt mark according to an embodiment of the present invention;

[0023] Figure 3 This is an image of the melting zone in an embodiment of the present invention;

[0024] Figure 4 This is a binary image from an embodiment of the present invention;

[0025] Figure 5 The image mask for semantic segmentation in this embodiment of the invention;

[0026] Figure 6 This is a difference image from an embodiment of the present invention;

[0027] Figure 7 The image is an example of an embodiment of the present invention after morphological erosion processing;

[0028] Figure 8 The image is the result of median filtering according to an embodiment of the present invention;

[0029] Figure 9 The image of the selected rectangular region in this embodiment of the invention;

[0030] Figure 10 This is an enlarged view of the grain region in an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0032] Embodiments of the present invention provide a metallographic image recognition method for wire weld marks based on semantic segmentation, such as... Figure 1 As shown, it includes the following steps:

[0033] Step 1: Preprocess the metallographic image of the conductor weld line. The processed image is shown below. Figure 2 As shown. Due to the inconsistent image size in the dataset, the metallographic images of wire melt marks were normalized to 512*512 pixels. Furthermore, to avoid image distortion during normalization, black pixels were used to maintain the image's aspect ratio.

[0034] Step 2: As Figure 2 As shown, since the metallographic image of the conductor melt mark is obtained directly after the conductor has undergone metallographic processing, the image contains the conductor area, the melted area, and other interfering parts. Different types of conductors are mainly distinguished by the characteristics of holes and grains, all of which are concentrated in the melted area. These contaminating factors, besides the melted area, cause significant deviations in image feature extraction. Therefore, it is necessary to extract the conductor melted area from the metallographic image of the conductor melt mark.

[0035] This embodiment uses the TransUnet neural network to perform semantic segmentation on the preprocessed metallographic image of the wire melt mark, obtaining an image of the melted area, such as... Figure 3 As shown, the TransUnet neural network is based on the traditional U-shaped network structure, combining CNN and Transformer, and has the advantages of both Transformers and U-Net. It combines the global self-attention mechanism of Transformer and the local spatial information recovery of U-Net network structure, and has achieved excellent results in semantic segmentation of metallographic images.

[0036] The TransUNet neural network consists of two parts: an encoder and a decoder. The encoder uses a traditional ResNetV2 convolutional neural network to extract features from the image. Then, the embedding module transforms the two-dimensional feature map into a one-dimensional feature sequence, achieving image feature serialization and transforming the image segmentation task into a sequence-to-sequence task. Finally, 12 consecutive Transformer modules transform the feature sequence into an output sequence with global context features. Furthermore, the multi-head attention mechanism in the Transformer modules allows the network to focus on the regions in the image features that need to be extracted; in this embodiment, this is the melting region. The decoder continuously upsamples the encoded feature sequence, combining it with convolutional neural network feature maps containing detailed image features. The decoder also incorporates U-Net's ability to repair spatial information in the image, ensuring accurate localization of the segmentation target.

[0037] Step 3: The OTSU algorithm is used to binarize the fused region image obtained from semantic segmentation, resulting in a binary image, such as... Figure 4As shown, the image can initially distinguish between wires and holes. The OTSU algorithm divides the image into two parts based on its grayscale characteristics: background and foreground (holes in the melted area). The larger the inter-class variance between the background and foreground, the greater the difference between the two parts of the image. Misclassifying part of the foreground as background or part of the background as foreground will reduce the difference between the two parts. The OTSU algorithm calculates the segmentation that maximizes the inter-class variance, i.e., minimizes the misclassification probability.

[0038] Using the image mask obtained from semantic segmentation in step 2, such as Figure 5 As shown, the difference image obtained by subtracting the image mask from the binary image is as follows. Figure 6 As shown, the difference image can roughly show the hole area in the metallographic image.

[0039] Because the image mask obtained from semantic segmentation has errors at the image segmentation boundaries, and because some areas of the melting region have similar gray levels to the holes, there are connected regions between the wire boundaries and the holes in the difference image, making it difficult to separate the holes effectively and extract the hole features accurately. Therefore, this embodiment uses morphological erosion to process the difference image. In mathematics, morphological erosion can eliminate object boundary points, remove objects smaller than the structuring element, and clear the small connections between two objects. This embodiment uses a 3x3 circular structuring pixel block, and iterates the morphological erosion on the difference image three times with the structuring pixel block to remove the connected regions between the holes, thus separating and extracting the holes. As shown in the formula:

[0040]

[0041] Where X is the set of pixels in the original image, and S is the set of pixel blocks in the structure.

[0042] The image obtained after morphological erosion of the difference image is as follows: Figure 7 As shown, redundant connected regions in the image are eliminated, but some discrete interference points still exist. Finally, median filtering is used to remove discrete points from the image, thus achieving the extraction of holes in the metallographic image of wire weld marks. Figure 8 The white area in the image represents the extracted hole area.

[0043] Calculate the 5-dimensional geometric features of the hole region, including average area, average perimeter, roundness, number, and percentage of total area.

[0044] Step 4: In this embodiment, the metallographic image of the wire melt mark contains a mixture of grain and void regions, making it difficult to locate and extract the grain region. To address this issue, the grain region is defined as the largest rectangular area in the melt zone image after removing the void region. Figure 7 The area within the white box is the largest rectangular region. Figure 8This is a magnified view of the grain region. Conventional iterative methods for extracting the grain region result in an average processing time of nearly one minute per image due to the high image resolution, which is insufficient for practical applications. Therefore, this embodiment designs a dynamic programming algorithm to extract the grain region, requiring only 5-7 seconds to process each image.

[0045] Grain characteristics are an important basis for distinguishing wire weld marks. Different types of wire weld marks contain different types of grains within the molten zone, such as equiaxed grains, eutectic grains, and columnar grains. Different types of grains have significant differences in texture characteristics, which can therefore be used as a classification criterion. In this embodiment, the texture features of the grain region in the preprocessed metallographic image of wire weld marks are extracted. The extracted texture features include GLCM texture features and LBP texture features.

[0046] GLCM texture features: 20-dimensional GLCM texture features including five attributes: energy, contrast, correlation, entropy, and homogeneity are calculated along four directions: 0°, 45°, 90°, and 135°.

[0047] Energy is the sum of the squares of the elements in the gray-level co-occurrence matrix, reflecting the uniformity of the gray-level distribution in an image. The more uniform the local gray-level distribution, the larger this value.

[0048] Contrast represents the sum of local variations in grayscale in an image, reflecting the image's sharpness. The higher the contrast, the clearer the texture.

[0049]

[0050] Correlation measures the similarity of image gray levels in a specific direction. It reflects the degree of local gray-level correlation. The higher the correlation, the larger the value.

[0051]

[0052] Entropy is a measure of the amount of information an image contains; it represents the degree of non-uniformity or complexity of texture in an image.

[0053]

[0054] Homogeneity reflects the homogeneity of image texture and measures the local variation of image texture.

[0055]

[0056] Where M is the gray-level co-occurrence matrix, (i,j) represents the coordinates of image pixels, and k represents the gray level of the image. Mean and Variance represent the mean and variance of the image's gray-level co-occurrence matrix, respectively.

[0057] LBP texture features: The LBP algorithm is used to encode the grain region to obtain an LBP texture image, and the number of each pixel in the LBP texture image is counted using a histogram as the LBP texture feature. In this embodiment, the LBP texture image is calculated using an LBP operator with a radius of r = 8, and then the feature values ​​of the LBP texture image are counted using a histogram to finally obtain a 10-dimensional LBP texture feature.

[0058] Step 5: First, based on the area of ​​the melt zone image, normalize the 5-dimensional geometric features of the hole region, the 10-dimensional LBP texture features and the 20-dimensional GLCM texture features of the grain region, and then input them into the classification model. After calculation, the classification model outputs the melt mark category.

[0059] This embodiment uses the XGBoost (eXtreme Gradient Boosting) classifier as the classification model to model the normalized wire weld line features. XGBoost is an additive model based on the boosting ensemble idea. It generates and constructs t CART regression trees, adding trees and continuously splitting during the iteration process to complete the tree construction. In each iteration, a CART tree is learned to fit the residual between the prediction results of the previous t-1 trees and the true values ​​of the training samples, thus forming a strong classifier.

[0060]

[0061] in f is the prediction result of sample i after the t-th iteration. t (x i Let be the model of the t-th tree.

[0062] After normalizing the 35-dimensional features using the XGBoost classifier, a model was established to automatically identify four types of metallographic images of conductor weld marks. In this embodiment, the weld mark categories include primary short-circuit weld marks, secondary short-circuit weld marks, equiaxed crystal weld marks caused by fire, and eutectic weld marks caused by fire.

[0063] The present invention has been described in detail above through embodiments, but the content described is only an exemplary embodiment of the present invention and should not be considered as limiting the scope of the present invention. The scope of protection of the present invention is defined by the claims. Any technical solutions designed by those skilled in the art using the technical solutions described in the present invention, or designed by those skilled in the art under the inspiration of the technical solutions of the present invention, within the substance and protection scope of the present invention, to achieve the above-mentioned technical effects, or any equivalent changes and improvements made to the scope of the application, should still fall within the patent protection scope of the present invention.

Claims

1. A metallographic image recognition method for wire weld marks based on semantic segmentation, characterized in that, Includes the following steps: Step 1: Preprocess the metallographic image of the wire melt mark; Step 2: Perform semantic segmentation on the preprocessed metallographic image of the wire melt mark to obtain the image of the melting zone in the melt mark feature set; Step 3: Perform binarization, morphological erosion, and median filtering on the image of the molten area in sequence to extract the hole region and calculate the geometric features of the hole region; Step 4: Select the largest rectangular region in the melt zone image after removing the hole region as the grain region, and calculate the LBP texture features and GLCM texture features of the grain region; Step 5: Input the geometric features of the hole region, the LBP texture features and GLCM texture features of the grain region into the classification model, and the classification model outputs the melt mark category.

2. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 1, preprocessing: the metallographic image of the wire melt mark is normalized to 512×512 pixels, and black pixels are used to ensure that the aspect ratio of the original image remains unchanged.

3. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 2, the TransUnet neural network is used to perform semantic segmentation on the preprocessed metallographic image of the wire melt mark.

4. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 3, the OTSU algorithm is used to binarize the melting area image to obtain a binary image. The difference between the semantic segmentation image mask in step 2 and the binary image is obtained. The difference image is iterated multiple times using circular structured pixel blocks in a morphological erosion manner to remove the connected regions between holes, and the holes are separated and extracted. Then, median filtering is used to remove discrete points in the image and extract the hole region.

5. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 3, the geometric features of the hole region include average area, average perimeter, roundness, number, and percentage of total area.

6. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 4, the LBP algorithm is used to encode the grain region to obtain the LBP texture image.

7. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 4, the GLCM texture features include energy, contrast, correlation, entropy, and homogeneity along four directions: 0°, 45°, 90°, and 135°.

8. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 5, the geometric features of the hole region, the LBP texture features and GLCM texture features of the grain region are normalized based on the area of ​​the melt zone image, and then input into the classification model.

9. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 5, the classification model is the XGBoost classifier.

10. The metallographic image recognition method for wire weld marks based on semantic segmentation as described in claim 1, characterized in that, In step 5, the melt marks are categorized as primary short-circuit melt marks, secondary short-circuit melt marks, equiaxed crystal melt marks from burning, and eutectic melt marks from burning.

Images