A steel surface defect detection method based on multi-modal features

Abstract

The application discloses a kind of steel surface defect detection methods based on multi-modal feature, comprising the following steps: obtaining the detection image of the steel surface to be detected;Load 3D defect atlas;3D defect atlas is used to integrate the defect category, position and parameter of steel surface;Load defect detection model, defect detection model adopts the LLaVA-BLIP2 cloud big model trained, combined with 3D defect atlas, according to visual feature and ultrasonic feature judges the defect information of detection steel surface;The detection image is input to the defect detection model, and the steel surface defect detection result is obtained.According to the above technical solution, the surface topography feature is captured using visual data, and the feature is complementary by combining ultrasonic information, which realizes the stereoscopic presentation of steel surface defect feature;Then a multi-modal big model is used to uniformly understand and extract features from heterogeneous data, forming a double insurance mechanism for defect discrimination, which significantly improves the detection capability of micro-defects and hidden defects and reduces the miss rate.

Description

Technical Field

[0001] This invention relates to the technical field of combining quality inspection with deep learning, and more specifically, to a method for detecting defects on steel surfaces based on multimodal features. Background Technology

[0002] In the steel manufacturing and processing industry, surface defects are one of the important factors affecting the quality of steel products. Common steel surface defects include cracks, porosity, inclusions, and scratches. These defects not only reduce the mechanical properties and corrosion resistance of steel, but also affect its appearance quality and reduce the product's market competitiveness.

[0003] Traditional steel surface defect detection mainly relies on manual visual inspection and simple image processing techniques. Manual visual inspection suffers from drawbacks such as high labor intensity, low efficiency, and susceptibility to subjective factors, making it difficult to meet the needs of large-scale steel production. Simple image processing techniques, on the other hand, can typically only detect some obvious defects, and are ineffective at detecting minute defects or defects against complex backgrounds, easily leading to missed or false detections.

[0004] With the development of artificial intelligence technology, deep learning-based image detection methods are gradually being applied to the field of steel surface defect detection. However, most existing deep learning-based methods only utilize single visual image information and do not fully consider the correlation between other relevant data (such as ultrasonic detection data) in the steel production process and surface defects, resulting in the need to further improve the accuracy and reliability of detection.

[0005] Therefore, there is an urgent need for a detection scheme that integrates multimodal large models and multimodal features to solve the problems of insufficient accuracy and reliability in steel surface defect detection and difficulty in dealing with complex scenarios. Summary of the Invention

[0006] To achieve the above objectives, this application provides a method for detecting surface defects in steel based on multimodal features, comprising the following steps: Acquire inspection images of the steel surface to be inspected; Load 3D defect maps; 3D defect maps are used to integrate the types, locations, and parameters of defects on the surface of steel bodies; Load the defect detection model, which uses a trained LLaVA-BLIP2 cloud-based large model and combines it with the 3D defect map to determine the defect information of the steel surface to be inspected based on visual and ultrasonic features. The detection image is input into the defect detection model to obtain the defect detection results on the steel surface.

[0007] Before loading the 3D defect map, visual features are extracted based on the annotated two-dimensional image of the measured steel surface, and a 3D defect map is established by combining the defect waveform data; the annotated two-dimensional image has marked the defect location and defect depth. The process of establishing a 3D defect map includes the following steps: Define the initial 3D space; the initial 3D space is a three-dimensional spatial coordinate system established by using a high-resolution image of defects on the steel surface as the reference plane and combining it with the defect depth information. A heat map is generated in the initial 3D space based on the visual features and defect depth information reflected in the 2D image; Based on the ultrasonic characteristics, the shape and size of the defect boundary are located, and the specific parameters of the defect are generated. A 3D defect map is generated based on the points and combination parameters corresponding to the three-dimensional coordinates of the defect features.

[0008] Among them, visual features extracted from annotated two-dimensional images of the measured steel surface include: Acquire raw images; The original image is subjected to contrast enhancement and image size unification processing to generate a high-resolution image of the defect; High-resolution images of defects are input into a pre-trained convolutional neural network model to extract visual features.

[0009] Furthermore, when generating the heatmap, for each location in the initial 3D space, a combination of three-dimensional coordinates of multiple points is generated based on their corresponding visual features and depth information to reflect the visual features of the steel surface; the combination parameters formed by each combination of three-dimensional coordinates reflect the defect features, including: defect type, defect location, depth, shape, and size; the defect probability value of the area corresponding to the combination of three-dimensional coordinates is calculated, and the value is mapped to different color ranges, with darker colors indicating a greater probability of a defect, thereby generating the heatmap.

[0010] Furthermore, the specific parameters for generating defects refer to: integrating ultrasonic features into the initial 3D space heatmap and correcting the heatmap values; if the ultrasonic features show inconsistent defect boundaries, updating the points and combination parameters of the three-dimensional coordinate combination.

[0011] The process of acquiring an inspection image of the steel surface to be inspected includes the following steps: Obtain the original high-resolution image; An adaptive histogram equalization algorithm is used to enhance the contrast and unify the image size of the original image to generate a detection image. The processing steps of the adaptive histogram equalization algorithm include: calculating the histogram of the original high-definition image, allocating the gray values ​​of the image pixels according to the calculation results, calculating the cumulative distribution function and mapping the pixel values ​​to enhance the image contrast, and then performing bilinear interpolation fusion to enhance the overall contrast of the image.

[0012] The LLaVA-BLIP2 cloud-based large model includes a visual encoder, a multimodal alignment module, and a language model in its structure. The visual encoder uses a pre-trained Vision Transformer to extract visual features from steel images and outputs a high-dimensional semantic vector. The multimodal alignment module is used to implement modal alignment; The language model uses the LLaMA-2 architecture and is responsible for generating defect descriptions or classification results.

[0013] The multimodal alignment module includes the BLIP2 path and the LLaVA path: The BLIP2 path uses Q-Former to compress visual features into a small number of query tokens, and interacts with text features through a cross-attention mechanism to achieve modality alignment; The LLaVA path uses a linear projection layer to directly map visual features to the embedding space of the language model, simplifying the structure and improving efficiency.

[0014] When the LLaVA-BLIP2 structure is combined with the 3D defect map, the 3D defect map is first preprocessed to align and normalize the visual features in space. Then, through the multimodal alignment module, the features of different modalities are mapped to a unified semantic space. The cross-attention mechanism is used to allow the features to interact dynamically and explore deep associations. After that, the map is divided into small regions by a sliding window and input into the model. With the help of weak supervision learning and contrastive learning, accurate localization and classification are achieved.

[0015] Furthermore, when implementing defect detection, the LLaVA-BLIP2 cloud-based large model uses an encoder-decoder architecture as its core, which includes: compressing the input data into a fixed-dimensional latent representation through the encoder, and then generating the target output based on the representation by the decoder; The encoder part receives 3D defect map data fused with visual features, and uses multi-layer convolution and attention mechanisms to deeply extract and encode key features in the data, explore potential correlations between different modalities, and form a high-dimensional semantic feature vector. The decoder adopts a dual-branch design, which is responsible for defect type classification and location regression prediction respectively. The branch responsible for defect type classification maps the encoded features to the probability distribution of different defect categories through fully connected layers and the Softmax activation function to achieve accurate classification. The branch responsible for location regression prediction uses fully connected layers to directly output the pixel-level coordinate information of steel surface defects.

[0016] This invention proposes a multimodal feature-based method for detecting defects on steel surfaces. It integrates visual images and ultrasonic data to construct a collaborative architecture combining deep learning and a large-scale multimodal model. This method utilizes visual data to capture surface morphology features and combines this with ultrasonic information to achieve feature complementarity, resulting in a three-dimensional representation of steel surface defect features. The large-scale multimodal model performs unified semantic understanding of heterogeneous data, while the deep learning network focuses on fine-grained feature extraction. The two enhance each other through feature alignment and decision fusion, forming a dual-insurance mechanism for defect identification. This invention effectively addresses the challenges of diverse defect morphologies and complex backgrounds, significantly improving the detection capability of minute and hidden defects, reducing the false negative rate, and simultaneously ensuring detection accuracy and system reliability. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the steps of a steel surface defect detection method based on multimodal features according to an embodiment of the present invention. Figure 2 This is a flowchart of adaptive histogram equalization provided according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the LLaVA-BLIP2 cloud-based large model structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the encoder-decoder data stream provided according to an embodiment of the present invention; Figure 5 This is a schematic diagram of ultrasonic feature detection provided according to an embodiment of the present invention. Detailed Implementation

[0018] The purpose of this invention is to provide a method for detecting defects on steel surfaces based on multimodal features and a large multimodal model. By fusing multimodal information such as visual images and ultrasonic data, and combining feature maps, a high-precision detection method for defects on steel surfaces can be achieved.

[0019] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] The present invention provides a method for detecting surface defects in steel based on multimodal features. Figure 1 As shown, it includes the following steps: Step S110: Obtain the inspection image of the steel surface to be inspected; In this invention, the detection of defects on the steel surface is based on high-definition images of the steel surface; the high-definition images are obtained through a high-resolution industrial camera.

[0021] After obtaining the original high-resolution image, an adaptive histogram equalization algorithm is used to enhance the contrast and unify the image size of the original image to generate a detection image.

[0022] The processing procedure of the adaptive histogram equalization algorithm is as follows: Figure 2 As shown: Histogram calculation is performed on the input high-definition image. Based on the calculation results, the grayscale values ​​of image pixels are allocated. Cumulative distribution function calculation and pixel value mapping are performed to enhance image contrast. Then, bilinear interpolation fusion is performed to enhance the overall contrast of the steel image and effectively improve image clarity. The adaptive histogram equalization algorithm is expressed as follows: ,in, r Represents the gray levels of the original image. s To output grayscale levels, L This represents the total number of gray levels. To represent the probability at this gray level, CDF is the cumulative distribution function.

[0023] In the bilinear interpolation fusion process, the original image is divided into multiple overlapping regions, and the histogram and cumulative distribution function are calculated independently for each region. The transformation functions of the four neighboring blocks are weighted and averaged, where the weights are inversely proportional to the distance from the pixel to the center of each block. The central region uses local mapping, while the boundary regions are smoothly blended with the mappings of adjacent blocks, thereby eliminating gray-level jumps between blocks and achieving local contrast enhancement and natural transition.

[0024] This step automatically adjusts the histogram distribution of local image regions to enhance contrast. Compared to traditional global histogram equalization, this method better preserves local image details. Simultaneously, to avoid increased image noise due to over-enhancement, a contrast limit threshold is set in the algorithm. When the contrast enhancement of a local region exceeds this threshold, the enhancement level is automatically limited.

[0025] Image size standardization refers to standardizing the image size according to the requirements of the subsequent model. For example, the image size is adjusted to a fixed 4:3 aspect ratio, and a bilinear interpolation algorithm is used for size transformation to ensure that the image does not exhibit obvious distortion and jagged edges during scaling.

[0026] Step S120: Load the 3D defect map; the 3D defect map is used to integrate the defect categories, locations and parameters of the steel body surface; Before loading the 3D defect map, visual features are extracted based on the annotated two-dimensional image of the measured steel surface, and a 3D defect map is established by combining the defect waveform data obtained by the ultrasonic flaw detector. The annotated two-dimensional image has marked the defect location and defect depth; the defect waveform data can reflect the defect location and other defect information.

[0027] Extracting visual features includes the following steps: 1) Acquire raw images: Use a high-resolution industrial camera with at least 20 megapixels to acquire raw images. The frame rate of the high-resolution industrial camera should be no less than 30 frames per second to ensure that steel images can be acquired in real time and accurately on high-speed production lines, and to avoid missing defect information due to untimely acquisition.

[0028] The original image's clarity and resolution capture extremely fine textures and potential defects on the steel surface, and is stored in a lossless compression format in a data storage device. The data storage device uses a solid-state drive array with high-speed read and write capabilities and large capacity, with read and write speeds reaching several GB per second. It can quickly store large amounts of image data and has a redundant backup mechanism to prevent data loss and ensure data integrity and security.

[0029] 2) Perform contrast enhancement and image size unification processing on the original image to generate a high-resolution image of the defects. The contrast enhancement and image size unification processing methods are the same as those in step S110.

[0030] 3) Input the high-resolution image of the defect into the pre-trained convolutional neural network model to extract visual features; among which, visual features include: texture, edge, and shape; the feature extraction method in this step can capture the visual features such as texture, edge, and shape of the steel surface, thereby reflecting the structural changes and potential defects inside the steel.

[0031] After visual feature extraction, a 3D defect map can be constructed by combining the defect waveform data at the corresponding location, including the following steps: Step S121: Define the initial 3D space; The initial 3D space is a three-dimensional spatial coordinate system established using a high-resolution image (two-dimensional image) of defects on the steel surface as the reference plane, combined with the detected defect depth information. In this system, pixels in the two-dimensional image correspond to their positions in three-dimensional space, with the x and y coordinates matching the pixel coordinates; the z coordinate is determined based on the defect condition: a default value (e.g., 0) is set for defect-free areas, while defective areas are assigned a value based on their actual depth, thus initially presenting the defect depth information.

[0032] Step S122: Generate a heat map in the initial 3D space based on the visual features and defect depth information reflected in the two-dimensional image; like Figure 5As shown, in this step, for each location in the initial 3D space, multiple points are combined into 3D coordinates based on their corresponding visual features (such as texture, edges, etc.) and depth information to represent the visual characteristics of the steel surface. The combination parameters formed by each 3D coordinate combination represent the defect characteristics. These parameters include: defect type (including types such as cracks, pores, inclusions, etc.), defect location, depth, shape, and size. Simultaneously, the defect probability value for the corresponding region of the 3D coordinate combination is calculated; a higher probability value indicates a higher probability of a defect. The values ​​are mapped to different color ranges, with darker colors indicating a higher probability of a defect, thus generating a heatmap.

[0033] Step S123: Based on the ultrasonic characteristics, locate the shape and size of the defect boundary and generate the specific parameters of the defect; Ultrasonic features, derived from defect waveform data, provide supplementary information such as the shape, size, and location of defects on the steel surface. This information is used to confirm the shape and size of defect boundaries, thus refining the defect representation. In this step, the ultrasonic features are integrated into the initial 3D space heatmap, and the heatmap values ​​are corrected: if the ultrasonic features indicate a wider or larger defect boundary, the points and combination parameters of the 3D coordinate combination are updated.

[0034] Step S124: Generate a 3D defect map based on the points and combination parameters of the three-dimensional coordinate combination corresponding to the defect features.

[0035] At this point, a 3D defect map is constructed based on visual and ultrasonic features. The 3D defect map integrates information such as the location, depth, shape, size, and probability of different defect categories (cracks, pores, inclusions, etc.) into three-dimensional space. The combination of points accurately locates and describes the detailed characteristics of defects, providing a comprehensive and intuitive display of surface and near-surface defects on the steel. For example, cracks appear as lines with a certain depth, width, and length; pores are voids with depth and diameter; and inclusions have specific shapes and distribution characteristics, providing rich information for subsequent defect location and type prediction.

[0036] Step S130: Load the defect detection model. The defect detection model uses a trained LLaVA-BLIP2 cloud-based large model, combined with a 3D defect map, to determine the defect information of the steel surface to be inspected based on visual and ultrasonic features. The LLaVA-BLIP2 cloud-based large model structurally includes a visual encoder, a multimodal alignment module, and a language model (LLM). 1) The visual encoder uses a pre-trained Vision Transformer to extract visual features from steel images and outputs a high-dimensional semantic vector; 2) The multimodal alignment module is used to implement modal alignment; specifically, it includes the BLIP2 path and the LLaVA path: The BLIP2 path uses Q-Former (Query Transformer) to compress visual features into a small number of query tokens and interacts with text features through a cross-attention mechanism to achieve modality alignment; the LLaVA path uses a linear projection layer to directly map visual features to the embedding space of the language model, simplifying the structure and improving efficiency. 3) The language model uses the LLaMA-2 architecture and is responsible for generating defect descriptions or classification results.

[0037] like Figure 3 As shown, the LLaVA-BLIP2 structure is not a single, directly integrated model, but rather combines the technical approaches of two multimodal large models, LLaVA and BLIP2. After extracting features through a visual encoder, an adapter is used to align the visual features with the language model, thereby achieving precise localization of tiny regions in the 3D defect map. This is represented as follows: , ; Language models are based on aligned visual features. and text instructions Generate response , is represented as: .

[0038] When the LLaVA-BLIP2 structure is combined with a 3D defect map, the 3D defect map is first preprocessed to spatially align and normalize visual features. Then, a multimodal alignment module maps features from different modalities to a unified semantic space, and a cross-attention mechanism allows features to interact dynamically, uncovering deep correlations. Afterward, a sliding window is used to divide the map into small regions, which are then input into the model. Through weakly supervised learning and contrastive learning, accurate localization and classification are achieved, fully integrating the advantages of multimodal approaches to improve the accuracy and reliability of steel surface defect detection.

[0039] The LLaVA-BLIP2 cloud-based large model uses an encoder-decoder architecture as its core for defect detection. The encoder-decoder architecture is as follows: Figure 4 As shown, the encoder compresses the input data into a fixed-dimensional latent representation (encoded vector), and the decoder generates the target output based on this representation. The encoding and decoding process is represented as follows: ,in, K For the number of defect categories, and The weights and biases for the classification branches.

[0040] In this structure, the encoder receives 3D defect map data that incorporates visual features, and uses multi-layer convolution and attention mechanisms to deeply extract and encode key features in the data, explore potential correlations between different modalities, and form a high-dimensional semantic feature vector; the decoder adopts a dual-branch design, which respectively undertakes the tasks of defect type classification and location regression prediction.

[0041] Among them, the branch responsible for defect type classification maps the encoded features to the probability distribution of different defect categories through fully connected layers and the Softmax activation function to achieve accurate classification; for example, it identifies cracks based on long strip-shaped and continuous edge features in the image; it determines the presence of pores based on optical and ultrasonic reflection features; and it distinguishes inclusions based on texture and ultrasonic attenuation characteristics; while the branch responsible for position regression prediction uses fully connected layers to directly output pixel-level coordinate information of steel surface defects.

[0042] During the training of the LLaVA-BLIP2 cloud-based large model, data augmentation techniques are employed to randomly rotate, translate, and scale images, simulating defect morphologies in different scenarios to improve the model's generalization ability. This ultimately forms the defect detection model after training. The training process is optimized by minimizing the weighted sum of classification and regression losses, as follows: ,in, For weighted systems; Regression loss; Classification loss.

[0043] Step S140: Input the detection image into the defect detection model to obtain the steel surface defect detection results.

[0044] In defect detection, the defect detection model, trained by the LLaVA-BLIP2 cloud-based large model, identifies defects in the detected images: the defect detection model can accurately predict the defect type and provide the location coordinates.

[0045] This invention presents specific detection cases and compares them with traditional SVM, CNN, and YOLOv7 methods. The steel surface defect detection method based on multimodal features provided by this invention outperforms traditional methods in both accuracy and false negative rate, as shown in Table 1. This demonstrates that the multimodal fusion mechanism provided by this invention effectively overcomes the limitations of single-modal methods, such as sensitivity to illumination and noise, and the inability to detect small targets. Furthermore, the strong generalization ability of the large model allows it to quickly adapt to different steel types and defect scenarios.

[0046] Table 1

[0047] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for detecting surface defects in steel based on multimodal features, characterized in that, Includes the following steps: Acquire inspection images of the steel surface to be inspected; Load a 3D defect map; the 3D defect map is used to integrate the defect categories, locations, and parameters of the steel body surface; Load the defect detection model, which uses a trained LLaVA-BLIP2 cloud-based large model and combines it with the 3D defect map to determine the defect information of the steel surface to be inspected based on visual and ultrasonic features. The detection image is input into the defect detection model to obtain the defect detection results on the steel surface.

2. The method for detecting steel surface defects based on multimodal features according to claim 1, characterized in that, Before loading the 3D defect map, visual features are extracted based on the annotated two-dimensional image of the measured steel surface, and a 3D defect map is established by combining the defect waveform data; the annotated two-dimensional image has marked the defect location and defect depth. The process of establishing a 3D defect map includes the following steps: Define an initial 3D space; the initial 3D space is a three-dimensional spatial coordinate system established by using a high-resolution image of defects on the steel surface as a reference plane and combining it with defect depth information. A heat map is generated in the initial 3D space based on the visual features and defect depth information reflected in the two-dimensional image; Based on the ultrasonic characteristics, the shape and size of the defect boundary are located, and the specific parameters of the defect are generated. A 3D defect map is generated based on the points and combination parameters corresponding to the three-dimensional coordinates of the defect features.

3. The method for detecting steel surface defects based on multimodal features according to claim 2, characterized in that, The extraction of visual features based on the annotated two-dimensional image of the measured steel surface includes: Acquire raw images; The original image is subjected to contrast enhancement and image size unification processing to generate a high-resolution image of the defect; The high-resolution image of the defect is input into a pre-trained convolutional neural network model to extract visual features.

4. The method for detecting steel surface defects based on multimodal features according to claim 2, characterized in that, When generating the heat map, for each location in the initial 3D space, a combination of three-dimensional coordinates of multiple points is generated based on their corresponding visual features and depth information to reflect the visual features of the steel surface. Each combination of three-dimensional coordinates constitutes a combination parameter that reflects the defect characteristics. The combination parameters include: defect type, defect location, depth, shape, and size. The defect probability value of the corresponding area of ​​the three-dimensional coordinate combination is calculated and mapped to different color intervals. The darker the color, the greater the probability of a defect, thereby generating a heat map.

5. The method for detecting steel surface defects based on multimodal features according to claim 4, characterized in that, The specific parameters for generating defects are: integrating ultrasonic features into the initial 3D space heatmap and correcting the heatmap values; if the ultrasonic features show inconsistent defect boundaries, updating the points and combination parameters of the three-dimensional coordinate combination.

6. The method for detecting steel surface defects based on multimodal features according to claim 1, characterized in that, The process of acquiring the inspection image of the steel surface to be inspected includes the following steps: Obtain the original high-resolution image; An adaptive histogram equalization algorithm is used to enhance the contrast and unify the image size of the original image to generate a detection image. The processing steps of the adaptive histogram equalization algorithm include: calculating the histogram of the original high-definition image, allocating the gray values ​​of the image pixels according to the calculation results, calculating the cumulative distribution function and mapping the pixel values ​​to enhance the image contrast, and then performing bilinear interpolation fusion to enhance the overall contrast of the image.

7. The method for detecting steel surface defects based on multimodal features according to claim 1, characterized in that, The LLaVA-BLIP2 cloud-based large model structurally includes a visual encoder, a multimodal alignment module, and a language model. The visual encoder uses a pre-trained Vision Transformer to extract visual features from the steel image and outputs a high-dimensional semantic vector. The multimodal alignment module is used to implement modal alignment; The language model uses the LLaMA-2 architecture and is responsible for generating defect descriptions or classification results.

8. The method for detecting steel surface defects based on multimodal features according to claim 7, characterized in that, The multimodal alignment module includes the BLIP2 path and the LLaVA path: The BLIP2 path uses Q-Former to compress visual features into a small number of query tokens, and interacts with text features through a cross-attention mechanism to achieve modality alignment; The LLaVA path uses a linear projection layer to directly map visual features to the embedding space of the language model, simplifying the structure and improving efficiency.

9. The method for detecting steel surface defects based on multimodal features according to claim 8, characterized in that, When the LLaVA-BLIP2 structure is combined with the 3D defect map, the 3D defect map is first preprocessed to align and normalize the visual features in space. Then, through the multimodal alignment module, features of different modalities are mapped to a unified semantic space. The cross-attention mechanism is used to enable dynamic interaction of features and uncover deep associations. After that, a sliding window is used to divide the small regions of the map into input models. With the help of weakly supervised learning and contrastive learning, accurate localization and classification are achieved.

10. The method for detecting steel surface defects based on multimodal features according to claim 8, characterized in that, When implementing defect detection, the LLaVA-BLIP2 cloud-based large model uses an encoder-decoder architecture as its core, including: compressing the input data into a fixed-dimensional latent representation through the encoder, and then generating the target output based on the representation by the decoder; The encoder part receives 3D defect map data with fused visual features, and uses multi-layer convolution and attention mechanisms to deeply extract and encode key features in the data, explore potential correlations between different modalities, and form a high-dimensional semantic feature vector. The decoder section adopts a dual-branch design, which respectively undertakes the tasks of defect type classification and location regression prediction. The branch responsible for defect type classification maps the encoded features to the probability distribution of different defect categories through a fully connected layer and a Softmax activation function to achieve accurate classification. The branch responsible for location regression prediction uses a fully connected layer to directly output the pixel-level coordinate information of steel surface defects.

Images