A metallographic structure analysis method based on a multi-task cooperative neural network architecture

Abstract

This invention discloses a metallographic structure analysis method based on a multi-task collaborative neural network architecture, belonging to the field of material microstructure analysis technology. Specifically, it includes the following steps: (1) acquiring raw image data; (2) performing data augmentation using various offline data enhancement methods; (3) constructing a multi-task collaborative neural network architecture, including a feature classification network based on deep residual optimization and an adaptive detection network with context awareness; (4) implementing a phased transfer training strategy, first training the classification network to achieve metal classification, then freezing its feature extraction layer parameters, and injecting classification semantic information into the detection network through a dynamic feature fusion module for end-to-end training; (5) based on the pixel-level segmentation results output by the detection network, using a connected component algorithm to achieve accurate area measurement of the metallographic structure. This method reduces model complexity through a parameter reuse mechanism. This method effectively solves the problems of background complexity sensitivity in traditional metallographic analysis, providing technical support for the intelligent development of metallography.

Description

Technical Field

[0001] This invention relates to the field of intelligent identification in metallographic analysis, specifically a metallographic structure analysis method based on a multi-task collaborative neural network architecture. Background Technology

[0002] Metallographic analysis plays a crucial role in industrial production. It not only helps us assess product quality and performance but also provides strong support for optimizing production processes and developing new products. Metallographic analysis is an indispensable stage in the production of alloy materials.

[0003] Current metallographic image analysis and detection methods include: traditional edge detection algorithms, morphological algorithms, optical flow algorithms, machine learning-based methods, and deep learning-based methods. As image complexity increases, the advantages of deep learning-based methods become increasingly apparent.

[0004] Most current deep learning methods are based on detection against a single background. However, when faced with complex backgrounds, existing algorithms struggle with foreground elements in one image category becoming blurred against the background of another, impacting detection accuracy. To address this issue, some research has split the task into two separate tasks: first classifying the images, then detecting each category separately. However, both the classification and detection networks in this approach involve feature extraction, leading to task overlap and redundancy in both time and space. Summary of the Invention

[0005] To address the aforementioned shortcomings and defects of existing technologies, this invention aims to propose a novel network structure for metallographic analysis based on a multi-task collaborative neural network architecture. This network structure enables the detection of metallographic structures in both simple and complex backgrounds. The novel network structure satisfies the following requirements: different networks are used for inference on different background images; and redundant network layers are avoided to prevent increasing the time and space complexity of the neural network. This effectively solves the problem of sensitivity to background complexity in traditional metallographic analysis, improving computational efficiency while maintaining model accuracy requirements. The specific technical solution is as follows:

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a metallographic structure analysis method based on a multi-task collaborative neural network architecture, comprising the following steps:

[0007] Images of various metals are acquired offline or in real time using image acquisition equipment and stored separately according to type.

[0008] The collected images were labeled according to the metallographic defects to be detected, and various offline image enhancement methods were used to increase the number and richness of samples. The resulting dataset was used for subsequent model training.

[0009] An end-to-end multi-task collaborative neural network model is constructed, including a metal type pre-selection submodule and a detection submodule that perform multi-task sequential connections for learning and identifying metal types and defect pixels; the metal type pre-selection submodule adopts a feature classification network based on deep residual optimization, and the detection submodule adopts an adaptive detection network with context awareness; and the number of branches in the detection submodule is consistent with the number of metal types in the image.

[0010] By using dataset samples to iteratively train and adjust the model parameters of the end-to-end multi-task collaborative neural network model, the performance of the model in detecting metallographic images with complex backgrounds is improved by minimizing the loss function.

[0011] In practical metallographic analysis applications, a trained ideal model is used to perform end-to-end multi-task processing on the metallographic image to be tested, outputting the type detection results and pixel segmentation results, and then returning to the original image to locate and calculate the area of ​​the defect target for further metallographic analysis.

[0012] The image acquisition process involves connecting a camera to a metallurgical microscope via a dedicated microscope interface, obtaining high-resolution metallographic images using the ideal light source of the metallurgical microscope, and then transmitting the acquired digital images to a computer via an image acquisition card, where they are classified and saved according to the type of metal.

[0013] The dataset was created by professional metallographic analysts who masked the defect targets in each type of metallographic structure image they collected, and then used at least one image enhancement method among horizontal flipping, vertical flipping, mirroring, contrast adjustment, brightness adjustment, histogram equalization, affine transformation, and adding salt-and-pepper noise to expand the dataset and improve the sample richness.

[0014] The metallographic types include various types of metals, alloys, ceramics, and LED chip packaging coatings; the defect target types include sulfides (A), alumina (B), silicates (C), spherical oxides (D), and large-particle spherical oxides (DS).

[0015] The structure of the feature classification network based on deep residual optimization includes a first Conv layer, a second Conv layer, a third C3 layer, a fourth Conv layer, a fifth C3 layer, a sixth Conv layer, a seventh C3 layer, an eighth Conv layer, a ninth C3 layer, and a tenth Classify layer connected in sequence; the outputs of the intermediate fifth layer, the seventh layer, the ninth layer, and the tenth layer are used as the network outputs.

[0016] The context-aware adaptive detection network consists of n identical blocks arranged in a block_list. The model structure of each block includes the following layers connected in sequence: the first CBAM layer, the second SPPF layer, the third PSA layer, the fourth upsampling layer, the fifth CBAcat layer, the sixth C3 layer, the seventh upsampling layer, the eighth CBAcat layer, the ninth C3 layer, the tenth Conv layer, the eleventh Concat layer, the twelfth C3 layer, the thirteenth Conv layer, the fourteenth Concat layer, the fifteenth C3 layer, and the sixteenth output layer. The eleventh layer is also connected to the sixth layer of this module, and the fourteenth layer is also connected to the third layer of this module.

[0017] The structural connection between the metal type pre-selection submodule and the detection submodule is as follows: the output layer of the metal type pre-selection submodule is connected to one of the blocks of the detection submodule based on the output result of the tenth layer. Specifically:

[0018] X′=model_cls(X)

[0019] Y=block_list[argmax(softmax(X′[-1]))](X′)

[0020] Where X represents the image input matrix, `model_cls` is the metal type pre-selection submodule, the output of the metal type pre-selection submodule X′, including the output of the fifth layer, the output of the seventh intermediate layer, the output of the ninth intermediate layer, and the output of the tenth layer. It also serves as the input of the detection submodule. The output of the tenth layer, after passing through the softmax layer, represents the probability of which of the n metal types the image belongs to, in the format (p1, p2, ..., p...). n The probability of a block is calculated by first argmax, then argmax is applied to the block list to determine which block in the list is used for detection. Only one block is used in each inference iteration. Y represents the final detection result, including the detection bounding box information (x, y, w, h) and the confidence information for each of the k target categories (C1, C2, ..., C...). k ), 32 mask extraction coefficients (m1, m2, ..., m 32 and mask feature module h m w represents the number of rows in the matrix of the mask feature modules. m This represents the number of columns in the matrix of the mask feature modules.

[0021] The calculation formula for the CBAM layer is as follows:

[0022] X′=σ(MLP(AveragePooling(X))+MLP(MaxPooling(X)))

[0023] Y=σ(Conv(Concat(Average(X′*X),Max(X′*X))))*X′

[0024] Where σ is the sigmoid function, MLP is the multilayer perceptron, AveragePooling is the average pooling, X is the input feature of this layer, MaxPooling is the max pooling, X′ represents the intermediate result of X after a series of processing, Y represents the output of this layer, Concat means concatenating two features, Average means calculating the average value, and Max means calculating the maximum value.

[0025] The formula for calculating the PSA layer is as follows:

[0026] A,B = Split(Conv(X))

[0027] Y=Conv(Concat(A,(B+Attn(B)+FFN(B+Attn(B)))))

[0028] Where Split means dividing a tensor into several sub-tensors, and FFN means a feedforward neural network;

[0029] The calculation formula for the Attn layer is as follows:

[0030] Q,K,V=Split(Conv(X))

[0031]

[0032] Where, d k The dimension of K is represented by Q, Q represents the information to be queried, K represents the vector being queried, and V represents the value obtained from the query.

[0033] The calculation formula for the CBAcat layer is as follows:

[0034] Y = Concat(X1, CBAM(X2))

[0035] Here, X1 and X2 represent the input features.

[0036] Furthermore, the fifth CBAcat layer in the block is also connected to the seventh layer of the metal type preselection submodule, and the eighth CBAcat layer is also connected to the fifth layer of the metal type preselection submodule.

[0037] Furthermore, the total number of parameters in the model is:

[0038] TP=CP+DP=CP+n*OBP=(1.16+n*1.18)M

[0039] Wherein, CP represents the number of parameters in the metal type pre-selection submodule, specifically 1.16M, DP represents the number of parameters in the detection submodule, OBP represents the number of parameters in a single block in the detection submodule, specifically 1.18M, ​​and n represents the number of blocks in block_list.

[0040] Furthermore, the total computational cost of the model in the method of this invention is:

[0041] TFs=CFs+DFs=CFs+n*OBFs=(1.86+n*1.9)GFLOPs

[0042] Wherein, CFs represents the computational cost of the metal type pre-selection submodule, specifically 1.86 GFLOPs, DFs represents the computational cost of the detection submodule, and OBFs represents the computational cost of a single block in the detection submodule, specifically 1.9 GFLOPs.

[0043] The model training process described is a phased transfer learning strategy. First, the classification network is trained to classify metals. Then, the parameters of its feature extraction layer are frozen. The classification semantic information is injected into the detection network through the dynamic feature fusion module for end-to-end training.

[0044] The metal type pre-selection submodule optimizer uses Adam, and the loss function is cross-entropy.

[0045] The detector submodule optimizer uses SGD, and the loss function consists of four parts: target loss, target bounding box regression loss, classification loss, and mask loss, with the output proportions allocated according to weights. The target loss is a softmax binary cross-entropy loss, the target bounding box regression loss is a CIou loss, the classification loss is a softmax cross-entropy loss, and the mask loss is a binary cross-entropy loss with logits.

[0046] The area of ​​the defect target is calculated as follows: the set B of pixel coordinates within the box in the search image whose pixel value is greater than the adjustable threshold in the mask confidence matrix. filt Calculate the product of the sum of the number of pixels at each position that are greater than the average pixel value and the square of the physical distance between the pixels.

[0047] The specific calculation formula is as follows:

[0048] a. Obtain the mask confidence matrix M;

[0049] M = Upsampe(Lm) T )

[0050] Where L represents the mask feature module, mT This is the transpose of the mask extraction coefficients;

[0051] b. Filter pixels with a confidence score greater than 0.6. Assume the set of pixel coordinates corresponding to the Box region in the image is B; select the set of pixel coordinates B whose values ​​in the mask confidence matrix M are greater than 0.6 from B. filt ;

[0052] B filt ={(x,y)∈B|M(x,y)>0.6}

[0053] c. For all in B filt For each pixel in image I, calculate the average grayscale value of that pixel in image I.

[0054]

[0055] Where I(x,y) represents the pixel value of the image at coordinates (x,y), |B filt | indicates B filt The total number of pixels in;

[0056] d. Filter out pixels within the Box region whose pixel values ​​are greater than the above average: Select pixels from the entire Box region B whose pixel values ​​I(x,y) are greater than the average value above ... The set of pixel coordinates, denoted as B final ;

[0057]

[0058] e. Calculate the total area of ​​the target area;

[0059] Area = |B final |×d 2

[0060] Among them, |B final | indicates B final The number of pixels in the array, where d represents the physical spacing between individual pixels. 2 This represents the physical area of ​​a single pixel.

[0061] The present invention has the following beneficial effects and advantages:

[0062] 1. This invention provides a novel neural network architecture. By combining a single classification network with multiple detection networks, a multi-task collaborative neural network architecture is proposed, which realizes end-to-end separate detection of multiple categories of images.

[0063] 2. While achieving classification before detection, the model's complexity is reduced due to redundancy of repeated modules.

[0064] 3. This invention provides a method for accurately calculating the area of ​​metallographic elements.

[0065] 4. The method of this invention effectively solves the problems of background complexity sensitivity in traditional metallographic analysis, and provides theoretical support for intelligent metallographic analysis research. Attached Figure Description

[0066] Figure 1 This is a flowchart of a metallographic structure analysis method based on a multi-task collaborative neural network architecture according to the present invention.

[0067] Figure 2 This is a schematic diagram of the end-to-end multi-task collaborative neural network architecture of the present invention.

[0068] Figure 3 This is a schematic diagram of the network structure of the CBAcat module.

[0069] Figure 4 This is a visual diagram illustrating the output results and area calculation of the metallographic image detection model of the present invention applied to a simple background.

[0070] Figure 5 This is a visual diagram illustrating the output results and area calculation of the metallographic image detection model of the present invention applied to complex backgrounds. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0072] The metallographic types that can be detected by the method of this invention include various types of metals and alloys, as well as ceramics and LED chip packaging coatings. For metals and alloys, detectable defects include sulfides (A), alumina (B), silicates (C), spherical oxides (D), and large-particle spherical oxides (DS). For images of ceramics and LED chip packaging coatings, the defect labels can be classified and labeled according to national standards. The method of this invention is used to increase the number of adaptive detection network modules and train and optimize the network model parameters, and then applied in practice to solve real-world problems.

[0073] like Figures 1-3 As shown, the metallographic structure analysis method based on a multi-task collaborative neural network architecture of the present invention includes the following steps:

[0074] Step 1) Image acquisition: Acquire images of various types of metals as the basis for building the dataset.

[0075] The image acquisition process involves using a camera that is directly connected to a metallurgical microscope via a dedicated microscope interface. High-resolution metallographic images are obtained using the ideal light source of the metallurgical microscope. The acquired digital images are then transmitted to a computer via an image acquisition device and classified and saved according to the type of metal.

[0076] Step 2) Dataset creation: Label the collected data with the targets to be detected, and use a variety of offline image enhancement methods to increase the number and richness of samples, making the trained model more robust. The final dataset will be used as the dataset for subsequent model training.

[0077] The dataset was created by professional metallographic analysts who performed mask annotations on the targets in each type of image they collected: First, the metallographic images with multiple backgrounds were classified according to metal type and saved to different folders in the same path. Second, the targets to be identified in each type of metallographic image were precisely masked.

[0078] Then, the dataset was expanded by using methods such as horizontal flipping, vertical flipping, mirroring, contrast adjustment, brightness adjustment, histogram equalization, affine transformation, and adding salt and pepper noise.

[0079] The dataset is divided into training, validation, and test sets in a ratio of 8:2:1. The training set data is used for iterative learning and training the model, the validation set data is used to verify the model's accuracy, and the test set data is used as instance images to be tested.

[0080] Step 3) Model Construction: Construct an end-to-end multi-task collaborative neural network model, such as... Figure 2 As shown.

[0081] The model structure includes a metal type pre-selection submodule and a detection submodule, with the number of branches in the detection submodule matching the number of image types. The metal type pre-selection submodule employs a feature classification network based on deep residual optimization, while the detection submodule uses an adaptive detection network with context awareness.

[0082] The relationship between the metal type pre-selection submodule and the detection submodule is such that the output layer of the metal type pre-selection submodule is connected to which block of the detection submodule, specifically, based on the output result of the tenth layer of the metal type pre-selection submodule.

[0083] X′=model_cls(X) (1)

[0084] Y=block_list[argmax(softmax(X′[-1]))](X′) (2)

[0085] Where X represents the image input matrix, `model_cls` is the metal type pre-selection submodule, the output of the metal type pre-selection submodule X′, including the output of the fifth layer, the output of the seventh intermediate layer, the output of the ninth intermediate layer, and the output of the tenth layer. It also serves as the input of the detection submodule. The output of the tenth layer, after passing through the softmax layer, represents the probability of which of the n metal types the image belongs to, in the format (p1, p2, ..., p...). n The probability of a metal category is then calculated using argmax, which determines which block in the block_list to use for detection. Only one block participates in the calculation for each inference. Y represents the final detection result of the model, including the detection bounding box information (x, y, w, h) and the confidence information of k target categories (C1, C2, ..., C...). k ), 32 mask extraction coefficients (m1, m2, ..., m 32 and mask feature module h m w represents the number of rows in the matrix of the mask feature modules. m This represents the number of columns in the matrix of the mask feature modules.

[0086] The structure of the feature classification network based on deep residual optimization is as follows: a first Conv layer, a second Conv layer, a third C3 layer, a fourth Conv layer, a fifth C3 layer, a sixth Conv layer, a seventh C3 layer, an eighth Conv layer, a ninth C3 layer, and a tenth Classify layer. The outputs of the intermediate fifth, seventh, ninth, and tenth layers are used as the network outputs.

[0087] Furthermore, the classification network has the following layer sizes: first layer with an input size of 640*640*3 and an output size of 128*128*16; second layer with an input size of 128*128*16 and an output size of 64*64*32; third layer with an input size of 64*64*32 and an output size of 64*64*32; fourth layer with an input size of 64*64*32 and an output size of 32*32*64; and fifth layer with an input size of 32*32*64 and an output size of 32*32. *64, the input size of the sixth layer is 32*32*64, and the output size is 16*16*128; the input size of the seventh layer is 16*16*128, and the output size is 16*16*128; the input size of the eighth layer is 16*16*128, and the output size is 8*8*256; the input size of the ninth layer is 8*8*256, and the output size is 8*8*256; the input size of the tenth layer is 8*8*256, and the output size is n*1, where n is the total number of background categories.

[0088] Since the output of the metal type pre-selection sub-model contains feature maps of all images, while the detection sub-module only needs features of specific metal images, CBAM, PSA, and CBAcat layers were designed to extract features of specific metal images. CBAM employs a lightweight attention module that improves network performance by explicitly modeling the importance of inter-channel and spatial information. The CBAM module consists of two parts: a channel attention module and a spatial attention module. The channel attention module performs average pooling and max pooling on the input data and sums the results, then uses convolutional layers and a sigmoid function to learn the importance of each channel. The spatial attention module calculates the average and maximum values ​​of the input data along the channel dimension and concatenates them, then uses convolutional layers and a sigmoid function to learn the importance of each pixel in the plane, finally multiplying them to obtain a weighted image feature representation.

[0089] The CBAM layer calculation formula is as follows:

[0090] X′=σ(MLP(AveragePooling(X))+MLP(MaxPooling(X))) (3)

[0091] Y=σ(Conv(Concat(Average(X′*X),Max(X′*X))))*X′ (4)

[0092] Where σ is the sigmoid function, MLP is a multilayer perceptron, AveragePooling is average pooling, X is the input feature of this layer, MaxPooling is max pooling, X′ represents the intermediate result of X after a series of processing, Y represents the output of this layer, Concat means concatenating two features, Average means calculating the average value, and Max means calculating the maximum value.

[0093] The formula for calculating the PSA layer is as follows:

[0094] A,B=Split(Conv(X)) (5)

[0095] Y=Conv(Concat(A,(B+Attn(B)+FFN(B+Attn(B)))))) (6)

[0096] Here, Split means dividing a tensor into several subtensors, and FFN represents a feedforward neural network.

[0097] The calculation formula for the Attn layer is as follows:

[0098] Q,K,V=Split(Conv(X)) (7)

[0099]

[0100] Where, d k The dimension of K is represented by Q, Q represents the information to be queried, K represents the vector being queried, and V represents the value obtained from the query.

[0101] A schematic diagram of the CBAcat layer is shown below. Figure 3 As shown, the calculation formula is as follows:

[0102] Y = Concat(X1, CBAM(X2)) (9)

[0103] Here, X1 and X2 represent the input features.

[0104] The detection submodule has a dynamically selected model structure, consisting of n blocks with the same structure combined into a block_list. The model structure of each block is as follows: first layer CBAM, second layer SPPF, third layer PSA, fourth layer upsampling, fifth layer CBAcat, sixth layer C3, seventh layer upsampling, eighth layer CBAcat, ninth layer C3, tenth layer Conv, eleventh layer Concat, twelfth layer C3, thirteenth layer Conv, fourteenth layer Concat, fifteenth layer C3, and sixteenth layer output.

[0105] The further details of the block are as follows: Layer 1 has an input size of 8*8*256 and an output size of 8*8*256; Layer 2 has an input size of 8*8*256 and an output size of 8*8*256; Layer 3 has an input size of 8*8*256 and an output size of 8*8*256; Layer 4 has an input size of 8*8*256 and an output size of 16*16*256; Layer 5 has an input size of 16*16*256 and an output size of 16*16*384; Layer 6 has an input size of 16*16*384 and an output size of 16*16*128; Layer 7 has an input size of 16*16*128 and an output size of 32*32*128; Layer 8 has an input size of 32*32*128 and an output size of 32*32*64 and an output size of 32*32*192; and Layer 9 has an input size of 32*32*192. *192, output size is 32*32*64; input size of the tenth layer is 32*32*64, output size is 16*16*64; input size of the eleventh layer is 16*16*64, 16*16*128, output size is 16*16*192; input size of the twelfth layer is 16*16*192, output size is 16*16*128; input size of the thirteenth layer is 16*16*128. The output size is 8*8*128. The input size of the fourteenth layer is 8*8*128, 8*8*256, and the output size is 8*8*384. The input size of the fifteenth layer is 8*8*384 and the output size is 8*8*256. The input size of the sixteenth layer is 32*32*64, 16*16*128, 8*8*256, and the output size is (32+5+n)*25200 and 32*160*160.

[0106] The connection layers are as follows: the fifth CBAcat layer in the block connects to the previous layer and the seventh layer of the metal type pre-selection submodule; the eighth CBAcat layer connects to the previous layer and the fifth layer of the metal type pre-selection submodule; the eleventh connection layer connects to the previous layer and the sixth layer; and the fourteenth connection layer connects to the previous layer and the third layer.

[0107] The total number of parameters in the model is

[0108] TP=CP+DP=CP+n*OBP=(1.16+n*1.18)M (10)

[0109] Wherein, CP represents the number of parameters in the metal type pre-selection submodule, specifically 1.16M, DP represents the number of parameters in the detection submodule, OBP represents the number of parameters in a single block in the detection submodule, specifically 1.18M, ​​and n represents the number of blocks in block_list.

[0110] The total number of parameters in a single inference iteration of the model using the method of this invention is:

[0111] OP = 1.16 + 1.18 = 2.34M (11)

[0112] If a traditional multi-task serial architecture using a non-parametric multiplexing mechanism that first classifies and then detects is adopted, the specific number of parameters is as follows:

[0113] The total number of parameters in the traditional method model is:

[0114] TP o =CP O +DP O =CP O +n*OBP O = (1.16 + n * 1.8)M (12)

[0115] Among them, CP O This represents the number of parameters in the classification model using traditional methods, specifically 1.16M, DP. O OBP represents the number of n detection model parameters using traditional methods. O This indicates the number of parameters for a single model in the detection submodule, specifically 1.8M.

[0116] The total number of parameters in a single inference iteration using traditional methods is:

[0117] OP O =1.16 + 1.8 = 2.96M (13)

[0118] The method of this invention not only reduces the number of parameters, but also reduces the amount of computation.

[0119] The total computational cost of the model in this invention is:

[0120] TFs=CFs+DFs=CFs+n*OBFs=(1.86+n*1.9)GFLOPs (14)

[0121] Wherein, CFs represents the computational cost of the metal type pre-selection submodule, specifically 1.86 GFLOPs, DFs represents the computational cost of the detection submodule, and OBFs represents the computational cost of a single block in the detection submodule, specifically 1.9 GFLOPs.

[0122] The total computational cost of a single inference iteration using the method of this invention is:

[0123] OFs=1.86+1.9=3.76GFLOPs (15)

[0124] The total computational cost of the traditional method model is:

[0125] TFs O =CFsO +DFs O =CFs O +n*OBFs O = (1.86 + n * 2.89) GFLOPs (16)

[0126] Among them, CFs O This represents the computational cost of the classification model using traditional methods, specifically 1.86 GFLOPs and DFs. O OBFs represents the computational cost of n detection models using traditional methods. O This indicates the computational cost of a single model in the detection submodule, specifically 2.89 GFLOPs.

[0127] The total computational cost of a single inference attempt using traditional methods is:

[0128] OFs O =1.86+2.89=4.75GFLOPs (17)

[0129] The total reduction in model parameters is

[0130] TP-TP O = (n*0.62)M (18)

[0131] When the number of detected sub-branches is 2, the percentage reduction in the total number of model parameters of the method of this invention compared with the traditional method is:

[0132]

[0133] The total reduction in model calculations is

[0134] TFs-TFs O = (n*0.99)GFLOPs (20)

[0135] When the number of detected sub-branches is 2, the percentage reduction in the total computational cost of the model using the method of this invention compared to the traditional method is:

[0136]

[0137] Step 4) Model training: Improve the model's performance in detecting metallographic images with complex backgrounds by adjusting the model parameters using data samples to minimize the loss function.

[0138] The model training process employs a phased transfer learning strategy. First, the classification network is trained to classify metals. Then, the parameters of its feature extraction layer are frozen, and the classification semantic information is injected into the detection network through a dynamic feature fusion module for end-to-end training.

[0139] The metal type pre-selection submodule optimizer uses Adam, and the loss function is cross-entropy.

[0140] The detector submodule optimizer uses SGD. The loss function consists of four parts: target loss, target bounding box regression loss, classification loss, and mask loss. The target loss is a softmax binary cross-entropy loss, the target bounding box regression loss is a CIou loss, the classification loss is a softmax cross-entropy loss, and the mask loss is a binary cross-entropy loss with logits.

[0141] Step 5) Model Application: Apply the model to the metallographic analysis system. The model outputs detection results and segmentation results. Calculate the target area using the model output and the original image for further metallographic analysis.

[0142] The model application involves applying the model to a metallographic analysis system, using the model's output to determine the category and area of ​​each target.

[0143] The target area is calculated as the product of the sum of the number of pixels whose pixel values ​​within the box in the image are greater than the average of pixels at positions greater than 0.6 in the mask confidence matrix, and the square of the physical distance between the pixels. The specific steps are as follows:

[0144] a. Obtain the mask confidence matrix M.

[0145] M = Upsampe(Lm) T ) (twenty two)

[0146] Where L represents the mask feature module, m T This is the transpose of the mask extraction coefficients.

[0147] b. Filter pixels with a confidence score greater than 0.6. Assume the set of pixel coordinates corresponding to the Box region in the image is B. Select the set of pixel coordinates from B whose values ​​in the mask confidence matrix M are greater than 0.6. filt .

[0148] B filt ={(x,y)∈B|M(x,y)>0.6} (23)

[0149] c. For all in B filt For each pixel in image I, calculate the average grayscale value of that pixel in image I.

[0150]

[0151] Where I(x,y) represents the pixel value of the image at coordinates (x,y), |B filt | indicates B filt The total number of pixels in the image.

[0152] d. Filter out pixels within the Box region whose pixel values ​​are greater than the above average: Select pixels from the entire Box region B whose pixel values ​​I(x,y) are greater than the average value above ... The set of pixel coordinates, denoted as B final .

[0153]

[0154] e. Calculate the total target area

[0155] Area = |B final |×d 2 (26)

[0156] Among them, |B final | indicates B final The number of pixels in the array, where d represents the physical spacing between individual pixels. 2 This represents the physical area of ​​a single pixel.

[0157] Experimental results:

[0158] like Figure 4 The diagram shows the output results and area calculation visualization of the metallographic image detection model of the present invention applied to a simple background. (a) is the original image, (b) is the mask result inferred by the model, (c) is the connected region calculated using the mask pixel value threshold, and (d) is the final accurate area of ​​the target. As can be seen from the diagram, for simple backgrounds, the present invention can effectively identify foreground targets from the background, such as simple backgrounds of stainless steel or alloy steel.

[0159] like Figure 5 The figure shows a visualization of the output and area calculation of the metallographic image detection model of the present invention applied to complex backgrounds. (a) is the original image, (b) is the mask result inferred by the model, (c) is the connected region calculated using the mask pixel value threshold, and (d) is the final accurate area of ​​the target. As can be seen from the figure, the present invention can effectively identify foreground targets from complex backgrounds. For example, the backgrounds of cobalt and nickel-based wrought superalloys are typically complex, as are some complex backgrounds of stainless steel and alloy steel.

[0160] The method of this invention can be applied to: analysis of metallic materials such as phase composition and heat treatment effect evaluation of metals and alloys; industrial testing such as coating quality of castings, LED chip packaging, and welding process verification; and research on the microscopic characteristics identification of ceramics, composite materials, and geological rock phases.

[0161] The above descriptions are merely specific embodiments of this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A metallographic structure analysis method based on a multi-task collaborative neural network architecture, characterized in that, Includes the following steps: Images of various metals are acquired offline or in real time using image acquisition equipment and stored separately according to type. The collected images were labeled according to the metallographic defects to be detected, and various offline image enhancement methods were used to increase the number and richness of samples. The resulting dataset was used for subsequent model training. An end-to-end multi-task collaborative neural network model is constructed, including a metal type pre-selection submodule and a detection submodule that perform multi-task sequential connections for learning and identifying metal types and defect pixels; the metal type pre-selection submodule adopts a feature classification network based on deep residual optimization, and the detection submodule adopts an adaptive detection network with context awareness; and the number of branches in the detection submodule is consistent with the number of metal types in the image. By using dataset samples to iteratively train and adjust the model parameters of an end-to-end multi-task collaborative neural network model, the performance of the model in detecting metallographic images with complex backgrounds is improved by minimizing the loss function. In practical metallographic analysis applications, a trained ideal model is used to perform end-to-end multi-task processing on the metallographic image to be tested, outputting the type detection results and pixel segmentation results, and then returning to the original image to locate and calculate the area of ​​the defect target for further metallographic analysis. The structure of the feature classification network based on deep residual optimization includes a first layer connected sequentially. Layer, second layer Layer 1, Layer 3 (C3), Layer 4 Layer 5, Layer C3, Layer 6 Layer 1, Layer 7, Layer C3, Layer 8 Layer 1, Layer 9 (C3), Layer 10 (Classify); The outputs of the fifth, seventh, ninth, and tenth intermediate layers serve as the network outputs. The context-aware adaptive detection network consists of n identical blocks arranged in a block list. The block's model structure includes, in sequence, a first CBAM layer, a second SPPF layer, a third PSA layer, a fourth upsampling layer, and a fifth layer. Layer 6, C3 layer, upsampling layer 7, layer 8 Layer 1, Layer 9, Layer C3, Layer 10 Floor, Eleventh Floor Floor 1, Floor 12 C3, Floor 13 The fourteenth floor Layer 1, Layer 15 (C3), Layer 16 (Output Layer); Layer 11 also connects to Layer 6 of this module, and Layer 14 also connects to Layer 3 of this module; The structural connection between the metal type pre-selection submodule and the detection submodule is as follows: the output layer of the metal type pre-selection submodule is connected to one of the blocks of the detection submodule based on the output result of the tenth layer. Specifically: ； ； Where X represents the image input matrix, , This is a submodule for pre-selecting metal types. The outputs of the metal type pre-selection submodule include the fifth-layer output, the seventh-layer intermediate output, the ninth-layer intermediate output, and the tenth-layer output. These also serve as the inputs to the detection submodule. The tenth-layer output result is processed... The layer after the image indicates the probability of which of the n metal classes the image belongs to, in the format of... After The value after represents the metal category with the highest probability, which determines which block in the block_list to use for detection. Only one block participates in the calculation for each inference. Y represents the final detection result of the model, including the detection bounding box information. Confidence information for k target categories 32 mask extraction coefficients and mask feature module , Let n be the number of rows in the matrix of the mask feature modules. This represents the number of columns in the matrix of the mask feature modules.

2. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The image acquisition process involves connecting a camera to a metallurgical microscope via a dedicated microscope interface, obtaining high-resolution metallographic images using the ideal light source of the metallurgical microscope, and then transmitting the acquired digital images to a computer via an image acquisition card, where they are classified and saved according to the type of metal.

3. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The dataset was created by professional metallographic analysts who masked the defect targets in each type of metallographic structure image they collected, and then used at least one image enhancement method among horizontal flipping, vertical flipping, mirroring, contrast adjustment, brightness adjustment, histogram equalization, affine transformation, and adding salt-and-pepper noise to expand the dataset and improve the sample richness.

4. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The metallographic types include various types of metals, alloys, ceramics, and LED chip packaging coatings; the defect target types include sulfides (A), alumina (B), silicates (C), spherical oxides (D), and large-particle spherical oxides (DS).

5. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The calculation formula for the CBAM layer is as follows: ； ； in, For the sigmoid function, It is a multilayer perceptron. For average pooling, These are the input features for this layer. For max pooling, This represents the intermediate result of X after a series of processing steps. This indicates the output of this layer. This indicates that two features are concatenated. This indicates calculating the average value. This indicates finding the maximum value; The formula for calculating the PSA layer is as follows: ； ； in, This indicates that the tensor is divided into several subtensors. This represents a feedforward neural network; The formula for calculating the layer is as follows: ； ； in, The dimension of K is represented by Q, Q represents the information to be queried, K represents the vector being queried, and V represents the value obtained from the query. The calculation formula for the CBAcat layer is as follows: ； in, and This represents the input features.

6. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The model training process described uses a phased transfer learning strategy. First, a classification network is trained to classify metals, and then the model is frozen. Its feature extraction layer parameters are used to inject classification semantic information into the detection network for end-to-end training through a dynamic feature fusion module; Among them, the metal type pre-selection submodule optimizer uses Adam, and the loss function uses cross-entropy; The detection submodule optimizer uses SGD, and its loss function consists of four parts: target loss, target bounding box regression loss, classification loss, and mask loss, with weights allocated to the output proportions. The target loss is... The binary cross-entropy loss, and the target bounding box regression loss are... Loss, classified as Cross-entropy loss, mask loss is with The binary cross-entropy loss.

7. The metallographic structure analysis method based on a multi-task collaborative neural network architecture according to claim 1, characterized in that, The area of ​​the defect target is calculated as follows: the set of pixel coordinates in the search image whose pixel value within the box is greater than the adjustable threshold in the mask confidence matrix. Calculate the product of the sum of the number of pixels at each position that are greater than the average pixel value and the square of the physical distance between the pixels; The specific calculation formula is as follows: a. Obtain the mask confidence matrix M; ； Where L represents the mask feature module, This is the transpose of the mask extraction coefficients; b. Filter pixels with a confidence score greater than 0.

6. Assume the set of pixel coordinates corresponding to the Box region in the image is B; select the set of pixel coordinates from B whose values ​​in the mask confidence matrix M are greater than 0.

6. ; ； c. For all in The pixels in the image are calculated. The average gray value in ; ； in, Indicates the image in coordinates Pixel value at that location, express The total number of pixels in; d. Filter out pixels within the Box region whose pixel values ​​are greater than the above average: Select those pixels from the entire Box region B. Greater than The set of pixel coordinates, denoted as ; ； e. Calculate the total target area; ； in, express The number of pixels in Represents the physical spacing of a single pixel. This represents the physical area of ​​a single pixel.

Images