Segmentation method, system and equipment for uneven metal surface defect feature distribution

Abstract

The invention discloses a segmentation method, system and equipment for uneven metal surface defect feature distribution. The segmentation method is a small sample segmentation method based on Graph Transform. Different from an existing Graph Transform based on graph nodes and edges, the Graph Transform provided by the invention has the advantages that a Patch-based graph embedding technology and a global-local graph-based Transform technology are utilized by the Graph Transform, the modeling is complicated, and the metal surface defect feature distribution is random. According to the method, the problems of complexity and randomness of metal surface defect feature distribution are solved through Graph Transform, a new solution is provided for actual segmentation of small sample metal surface defects, the segmentation performance and analysis results can better meet task requirements, and the method has good application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of metal surface defect detection, and particularly to a segmentation method, system and device for uneven distribution of metal surface defect features. Background Art

[0002] In recent years, due to the scarcity of metal surface defect samples in industrial application scenarios and the high cost of manual annotation, small sample segmentation schemes have received extensive attention. However, the defect features in metal surface defect samples may be unevenly distributed, which means that the features such as the position, size, and shape of these defects in the metal samples do not have obvious regularities, randomly appear at various positions in the samples, and there may be complex correlation relationships between different defects. For example, some defects may co-occur with other defects, or the existence of one defect will affect the formation and development of another defect. This randomness and complexity make traditional small sample detection methods have high computational complexity and are difficult to accurately identify and understand these defects. Summary of the Invention

[0003] Aiming at the above problems, the purpose of the present invention is to propose a segmentation method, system and device for alleviating the uneven distribution of metal surface defect features.

[0004] In order to achieve the above technical purpose, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a segmentation method for uneven distribution of metal surface defect features. This method is a small sample metal surface defect segmentation algorithm based on Graph Transformer. The Graph Transformer uses Patch-based graph embedding technology and global-local graph-based Transformer technology to model the complex and random distribution of metal surface defect features; The Patch-based graph embedding technology divides the input features into multiple patches of equal size through convolution, dimensionality reduction, and linear activation operations, and then embeds them into the graph structure to achieve dense prediction of defects and linear-level calculation; The global-local graph-based Transformer technology embeds the global graph and local graph of the support-query pair into (query, key, value) to implement the self-attention mechanism; The self-attention mechanism calculates the attention scores under different patches through the (query, key) of the local graph, and jointly acts on the (value) of the local graph and the (query, key, value) of the global graph to achieve attention to the defect topology structure.

[0005] As a possible implementation, further, the global graph obtains the global topological structure of the metal surface defect samples by embedding the Patches of the support-query pairs into the graph structure; the local graph obtains the local topological structure of the samples by embedding the masked support Patches and query Patches into the graph structure; The support-query pair is the task setting in the few-shot meta-learning framework; the (query, key, value) belongs to the Transformer framework and is used to calculate the attention scores.

[0006] As a possible implementation, further, the meta-learning framework sets the input data as an N-way K-shot task and divides it into a support set and a query set, simulating a small amount of labeled data in the real task to guide the model training; Among them, N-way means that the input data contains N categories, and K-shot means that each category contains K images; the support set is the sample set for model training and learning, and the query set is the sample set for testing and evaluating the model performance.

[0007] As a possible implementation, further, the task setting of the few-shot segmentation algorithm is 1-way K-shot, and it adopts an end-to-end training method, and sets a binary cross-entropy loss and a graph structure alignment loss in the downstream task:

[0008] Among them, is the binary cross-entropy loss, and the binary cross-entropy loss guides the gradient update of the model by evaluating the distance between the predicted label and the true label; is the graph structure alignment loss, and the graph structure alignment loss assists in guiding the gradient update of the model by evaluating the difference in the alignment operations during the image embedding Patch process and the Patch embedding graph structure process; is a hyperparameter.

[0009] As a possible implementation, further, the few-shot metal surface defect segmentation algorithm based on Graph Transformer includes a Patch encoder, a graph encoder, a Graph Transformer, and a residual decoder; An FPN encoder and a Patch embedding layer are provided on the Patch encoder; The FPN encoder encodes the 1-way K-shot support-query pair samples into 1 query feature and 1 support feature; The Patch embedding layer divides into query Patches, support Patches, and masked support Patches of equal size through a scale linear activation layer using query features, support features, and the masks corresponding to the support features; The graph encoder fuses the query Patches and support Patches and encodes them into the global graph through the graph transformation layer. At the same time, the graph encoder fuses the query Patches and masked support Patches and encodes them into the local graph through the graph transformation layer. Then, the features of the global graph and local graph are extracted through the graph convolutional network respectively; The global graph fuses with the query Patches using the complete support Patches and can represent the topological structure of the global distribution; the local graph fuses with the query Patches using the masked support Patches and can represent the topological structure of the local defect distribution; the global graph and local graph regard each Patch as a node, the channel features of each Patch as the information contained in the node, and the relative position relationship between Patches as edge weights; The Graph Transformer embeds the global graph and local graph into the multi-head attention based on the (query, key, value) paradigm, calculates the attention scores corresponding to the defect topological structure, and injects them into the global graph and local graph respectively; The residual decoder is provided with a mapping layer and a residual decoding layer; the mapping layer maps the global graph and local graph passing through the Graph Transformer from the graph space back to the feature space to obtain the global feature and local feature; among them, the graph space is used to describe the space where the Graph Transformer is located; the feature space is used to describe the space where the Patch encoder, graph encoder, and residual decoder are located; The residual decoding layer fuses the global feature, local feature, and query feature and decodes them into the final prediction result for calculating the binary cross-entropy loss; the residual decoding layer also separately decodes the local feature to obtain the intermediate prediction result for calculating the graph structure alignment loss.

[0010] As a possible implementation, further, the sizes of the query Patches, support Patches, and masked support Patches are controlled by hyperparameters to ensure that their widths and heights can be divisible by the width and height of the support-query pair features; the width and height of the support-query pair features are determined by the size of the input image and the FPN encoder.

[0011] As a possible implementation, further, the binary cross-entropy loss reduces the difference between the true distribution and the predicted distribution and can be expressed as:

[0012] Among them, N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples; the graph structure alignment loss can reduce the difference during the transformation of features between the graph space and the feature space, and can be expressed as:

[0013] Among them, N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples.

[0014] In a second aspect, the present invention provides a metal surface defect detection system, including a pipeline sample management system and a data analysis system; The pipeline sample management system uses 5G wireless technology and high-speed industrial cameras to collect and transmit sample images in real time for the data analysis system to process; The data analysis system is loaded with the above-mentioned segmentation method for uneven distribution of metal surface defect features for processing input data and exporting corresponding analysis reports; The analysis report includes visual images of the segmentation results, detailed parameters of each segmentation region (such as area, perimeter, shape features, etc.), segmentation accuracy evaluation indicators (such as Dice coefficient, Jaccard index, etc.), as well as statistical analysis and trend prediction based on the segmentation results.

[0015] In a third aspect, the present invention provides a metal surface defect detection method, including the following steps: S101. Obtain metal surface sample images on the production line through a high-speed industrial camera and transmit them to the industrial control computer through 5G wireless technology; S102. The industrial control computer is loaded with the above-mentioned segmentation method for uneven distribution of metal surface defect features, and performs real-time segmentation on the input defect samples and transmits the segmentation results to the data analysis system; S103. The data analysis system generates an analysis report containing visual images, detailed parameters of each segmentation region, segmentation accuracy evaluation indicators, as well as statistical analysis and trend prediction based on the segmentation results for industrial production management.

[0016] In a fourth aspect, the present invention further provides an electronic device, which includes: One or more processors, a memory, a graphics card, and one or more programs; Among them, the one or more programs are stored in the memory and are configured to be executed by the one or more processors and the graphics card, and the one or more programs can be used to execute the above-mentioned segmentation method for uneven distribution of metal surface defect features.

[0017] Adopting the above technical solutions, compared with the prior art, the present invention has the following beneficial effects: The present invention solves the problems of complexity and randomness in the feature distribution of metal surface defects through a few-shot metal surface defect segmentation algorithm based on Graph Transformer, provides a new solution for the actual implementation of few-shot metal surface defect segmentation, and enables the segmentation performance and analysis results to better meet the task requirements.

[0018] The few-shot segmentation algorithm for uneven feature distribution of metal surface defects provided by the present invention realizes dense prediction and linear-level calculation of metal surface defects through the processing of support-query pair samples by a Patch encoder and a graph encoder applicable to the meta-learning paradigm, and the reprocessing of a Graph Transformer and a residual decoder; the graph embedding method based on Patches can model the complex and random distribution of metal surface defect features; the GraphTransformer combining the global graph and the local graph realizes the synergistic effect of the global and local defect distributions and can handle the complex and random topological structures of metal surface defect features; in addition, through the graph alignment loss and the residual decoder applicable to mapping the global graph and the local graph to the feature space and completing the final prediction, the gradient calculation and backpropagation of the model can be optimized.

[0019] By designing a metal surface defect detection system, method and an electronic device, and combining with the few-shot segmentation algorithm, the present invention can realize the full-process management of metal surface defect detection on the industrial assembly line, improve the detection efficiency, and greatly reduce the labor cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0021] Figure 1 It is a schematic diagram of the data processing flow of the Patch encoder in the present invention; Figure 2 It is a schematic diagram of the data processing flow of the graph encoder in the present invention; Figure 3 It is a schematic diagram of the data processing flow of the Graph Transformer in the present invention; Figure 4 It is a schematic diagram of the data processing flow of the residual decoder in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0022] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be specifically noted that the following embodiments are only used to illustrate the present invention, but do not limit the scope of the present invention. Similarly, the following embodiments are only partial embodiments of the present invention rather than all embodiments. All other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

[0023] Embodiment 1 To alleviate the complexity and randomness of the distribution of metal surface defect features, this embodiment provides a few-shot segmentation method based on Graph Transformer, which is a few-shot metal surface defect segmentation algorithm based on Graph Transformer. Different from the existing Graph Transformer based on graph nodes and edges, the Graph Transformer in this embodiment uses Patch-based graph embedding technology and global-local graph-based Transformer technology to model the complex and random distribution of metal surface defect features.

[0024] In this embodiment, the Patch-based graph embedding technology divides the input features into multiple patches of equal size through convolution, dimensionality reduction, and linear activation operations, and then embeds them into the graph structure to achieve dense prediction of defects and linear-level calculations.

[0025] In this embodiment, the global-local graph-based Transformer technology embeds the global graph and local graph of the support-query pair into (query, key, value) to implement the self-attention mechanism; where the support-query pair is a task setting in the few-shot meta-learning framework, and (query, key, value) belongs to the Transformer framework and is used to calculate the attention score; the global graph obtains the global topological structure of the metal surface defect samples by embedding the patches of the support-query pair into the graph structure; the local graph obtains the local topological structure of the sample by embedding the masked support patches and query patches into the graph structure.

[0026] The self-attention mechanism calculates the attention scores under different patches through the (query, key) of the local graph, and jointly acts on the (value) of the local graph and the (query, key, value) of the global graph to achieve attention to the defect topological structure.

[0027] In this embodiment, the meta-learning framework sets the input data as an N-way K-shot task and divides it into a support set and a query set, simulating a small amount of labeled data in a real task to guide the model training. Here, N-way indicates that the input data contains N categories, K-shot means that each category contains K images, the support set is a sample set for model training and learning, and the query set is a sample set for testing and evaluating the model performance.

[0028] Similar to the existing methods, the task of the few-shot segmentation algorithm in this embodiment is set as 1-way K-shot, and an end-to-end training method is adopted; the difference is that in this embodiment, a binary cross-entropy loss and a graph structure alignment loss are set in the downstream task:

[0029] In the above formula, is the binary cross-entropy loss, is the graph structure alignment loss, is a hyperparameter. Among them, the binary cross-entropy loss guides the gradient update of the model by evaluating the distance between the predicted label and the true label; the graph structure alignment loss assists in guiding the gradient update of the model by evaluating the difference in the alignment operations during the image embedding Patch process and the Patch embedding graph structure process.

[0030] In this embodiment, the few-shot metal surface defect segmentation algorithm based on Graph Transformer includes a Patch encoder 100, a graph encoder 110, a Graph Transformer 120, and a residual decoder 130.

[0031] Referring to the appendix Figure 1 As shown, an FPN encoder 101 and a Patch embedding layer 102 are provided on the Patch encoder 100; the FPN encoder 101 encodes the 1-way K-shot support-query pair samples into 1 query feature 103 and 1 support feature 104, and the Patch embedding layer 102 uses the query feature 103, the support feature 104, and the mask 105 corresponding to the support feature, and divides them into equal-sized query Patches 107, support Patches 108, and masked support Patches 109 through a scale linear activation layer 106. The sizes of the query Patches 107, support Patches 108, and masked support Patches 109 are controlled by hyperparameters to ensure that their widths and heights can be divisible by the widths and heights of the support-query pair features. The widths and heights of the support-query pair features are determined by the size of the input image and the FPN encoder 101.

[0032] Referring to the appendix Figure 2As shown in the figure, the graph encoder 110 fuses the query Patches 107 and the support Patches 108, and encodes them into the global graph 112 through the graph transformation layer 111; at the same time, the graph encoder 110 fuses the query Patches 107 and the masked support Patches 109, and encodes them into the local graph 113 through the graph transformation layer 111; then, the graph convolutional network is used to extract the features of the global graph 112 and the local graph 113 respectively. Among them, the global graph 112 fuses the complete support Patches 107 with the query Patches 107, and can represent the topological structure of the global distribution; the local graph 113 fuses the masked support Patches 109 with the query Patches 107, and can represent the topological structure of the local defect distribution; the global graph 112 and the local graph 113 regard each Patch as a node, the channel feature of each Patch as the information contained in the node, and the relative position relationship between Patches as the edge weight.

[0033] Refer to the appendix Figure 3 As shown in the figure, the Graph Transformer 120 embeds the global graph 112 and the local graph 113 into the multi-head attention 121 based on the (query, key, value) paradigm, calculates the attention scores 122 corresponding to the defect topological structure, and injects them into the global graph 112 and the local graph 113 respectively.

[0034] Refer to the appendix Figure 4 As shown in the figure, the residual decoder 130 is provided with a mapping layer 131 and a residual decoding layer 132; the mapping layer 131 maps the global graph 112 and the local graph 113 passing through the Graph Transformer 120 from the graph space 133 back to the feature space 134, obtaining the global feature 135 and the local feature 136. Among them, the graph space 133 is used to describe the space where the Graph Transformer 120 is located; the feature space 134 is used to describe the space where the Patch encoder 100, the graph encoder 110, and the residual decoder 130 are located.

[0035] The residual decoding layer 132 fuses the global feature 135, the local feature 136, and the query feature 103, and decodes them into the final prediction result 137 for calculating the binary cross-entropy loss; the residual decoding layer 132 also separately decodes the local feature 136 to obtain the intermediate prediction result 138 for calculating the graph structure alignment loss.

[0036] Among them, the binary cross-entropy loss reduces the difference between the true distribution and the predicted distribution, and can be expressed as:

[0037] In the above formula, N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples. The graph structure alignment loss can reduce the difference during the transformation of features between the graph space and the feature space, and can be expressed as:

[0038] Among them, N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples.

[0039] Embodiment 2 This embodiment provides a metal surface defect detection system, including a pipeline sample management system and a data analysis system; The pipeline sample management system uses 5G wireless technology and high-speed industrial cameras to collect and transmit sample images in real time for the data analysis system to process; The data analysis system is loaded with the segmentation method for uneven distribution of metal surface defect features in Embodiment 1 for rapid processing of input data and exporting corresponding analysis reports; among them, the analysis report includes visual images of the segmentation results, detailed parameters of each segmentation region (such as area, perimeter, shape features, etc.), segmentation accuracy evaluation indicators (such as Dice coefficient, Jaccard index, etc.), and statistical analysis and trend prediction based on the segmentation results.

[0040] Embodiment 3 This embodiment provides a metal surface defect detection method, including the following steps: S101. The pipeline sample management system obtains metal surface sample images on the production line through high-speed industrial cameras and transmits them to the industrial control computer through 5G wireless technology; S102. The industrial control computer is loaded with the segmentation method for uneven distribution of metal surface defect features in Embodiment 1, and real-time segments the input defect samples and transmits the segmentation results to the data analysis system; S103. The data analysis system generates an analysis report containing visual images, detailed parameters of each segmentation region, segmentation accuracy evaluation indicators, and statistical analysis and trend prediction based on the segmentation results for industrial production management.

[0041] Embodiment 4 Based on the same inventive concept, this embodiment provides an electronic device, which includes: One or more processors, a memory, a graphics card, and one or more programs; Among them, one or more programs are stored in the memory and are configured to be executed by one or more processors and the graphics card, and one or more programs can be used to execute the segmentation method for uneven distribution of metal surface defect features in Embodiment 1.

[0042] The above are only some embodiments of the present invention, and thus do not limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall similarly be included in the patent protection scope of the present invention.

Claims

1. A segmentation method for uneven distribution of metal surface defect features, characterized in that, This method is a few-shot metal surface defect segmentation algorithm based on GraphTransformer. The Graph Transformer uses patch-based graph embedding technology and global-local graph-based Transformer technology to model the complex and random distribution of metal surface defect features; The patch-based graph embedding technology divides the input features into multiple patches of equal size through convolution, dimensionality reduction, and linear activation operations, and then embeds them into the graph structure; The global-local graph-based Transformer technology embeds the global graph and local graph of the support-query pair into queries, keys, and values to implement the self-attention mechanism; The self-attention mechanism calculates the attention scores under different patches through the queries and keys of the local graph, and jointly acts on the values of the local graph and the queries, keys, and values of the global graph to achieve attention to the defect topology; 2. The segmentation method for uneven distribution of metal surface defect features according to claim 1, characterized in that, The global graph obtains the global topology of the metal surface defect samples by embedding the patches of the support-query pair into the graph structure; the local graph obtains the local topology of the samples by embedding the masked support patches and query patches into the graph structure; The support-query pair is the task setting in the few-shot meta-learning framework; the queries, keys, and values belong to the Transformer framework and are used to calculate attention scores; 3. The segmentation method for uneven distribution of metal surface defect features according to claim 2, characterized in that The meta-learning framework sets the input data as an N-way K-shot task and divides it into a support set and a query set to simulate a small amount of labeled data in a real task and guide the model training; Among them, N-way means that the input data contains N categories, and K-shot means that each category contains K images; the support set is a sample set for model training and learning, and the query set is a sample set for testing and evaluating the model performance; 4. The segmentation method for uneven distribution of metal surface defect features according to claim 3, characterized in that, The task setting of the few-shot segmentation algorithm is 1-way K-shot, and it adopts an end-to-end training method, and sets a binary cross-entropy loss and a graph structure alignment loss in the downstream task: Among them, is the binary cross-entropy loss, which guides the gradient update of the model by evaluating the distance between the predicted label and the true label; is the graph structure alignment loss, which assists in guiding the gradient update of the model by evaluating the difference in the alignment operations during the process of embedding image patches and the process of embedding patches into the graph structure; is a hyperparameter.

5. The segmentation method for uneven distribution of metal surface defect characteristics according to claim 4, wherein The few-shot metal surface defect segmentation algorithm based on GraphTransformer includes a Patch encoder, a graph encoder, a Graph Transformer, and a residual decoder; An FPN encoder and a Patch embedding layer are provided on the Patch encoder; The FPN encoder encodes the 1-way K-shot support-query pair samples into 1 query feature and 1 support feature; The Patch embedding layer divides the query feature, support feature, and the mask corresponding to the support feature into equal-sized query patches, support patches, and masked support patches through a scale linear activation layer; The graph encoder fuses query Patches and support Patches and encodes them into the global graph through the graph transformation layer. At the same time, the graph encoder fuses query Patches and masked support Patches and encodes them into the local graph through the graph transformation layer. Then, the features of the global graph and the local graph are extracted through the graph convolutional network respectively; In the global graph and the local graph, each Patch is regarded as a node, the channel feature of each Patch is regarded as the information contained in the node, and the relative position relationship between Patches is regarded as the edge weight; The Graph Transformer embeds the global graph and the local graph into the multi-head attention based on the query, key, and value paradigm, calculates the attention scores corresponding to the defect topology structure, and injects them into the global graph and the local graph respectively; The residual decoder is provided with a mapping layer and a residual decoding layer; the mapping layer maps the global graph and the local graph passing through the Graph Transformer from the graph space back to the feature space to obtain the global feature and the local feature; among them, the graph space is used to describe the space where the Graph Transformer is located; the feature space is used to describe the space where the Patch encoder, the graph encoder, and the residual decoder are located; The residual decoding layer fuses the global feature, the local feature, and the query feature, and decodes them into the final prediction result for calculating the binary cross-entropy loss; the residual decoding layer also separately decodes the local feature to obtain the intermediate prediction result for calculating the graph structure alignment loss.

6. The segmentation method for uneven distribution of metal surface defect features according to claim 5, characterized in that, The sizes of the query Patches, the support Patches, and the masked support Patches are controlled by hyperparameters to ensure that their widths and heights can be divisible by the width and height of the support-query pair features; the width and height of the support-query pair features are determined by the size of the input image and the FPN encoder.

7. The segmentation method for uneven distribution of metal surface defect features according to claim 5, characterized in that The binary cross-entropy loss reduces the difference between the true distribution and the predicted distribution, and can be expressed as: where N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples; the graph structure alignment loss can reduce the difference during the transformation of features between the graph space and the feature space, and can be expressed as: where N represents the number of samples, y represents the true distribution of the samples, and p represents the predicted distribution of the samples.

8. A metal surface defect detection system, characterized in that, It includes a pipeline sample management system and a data analysis system; The pipeline sample management system uses 5G wireless technology and high-speed industrial cameras to collect and transmit sample images in real time for the data analysis system to process; The data analysis system loads the segmentation method for uneven distribution of metal surface defect features as described in any one of claims 1 to 7 for processing the input data and exporting the corresponding analysis report; The analysis report includes the visualization image of the segmentation result, the detailed parameters of each segmentation region, the segmentation accuracy evaluation index, and the statistical analysis and trend prediction based on the segmentation result.

9. A method for detecting metal surface defects, characterized in that, It includes the following steps: S101. Obtain the metal surface sample image on the production line through a high-speed industrial camera and transmit it to the industrial control computer through 5G wireless technology; S102. The industrial control computer is loaded with the segmentation method for uneven distribution of metal surface defect features described in any one of claims 1 to 7, and the input defect samples are segmented in real time, and the segmentation results are transmitted to the data analysis system; S103. The data analysis system generates an analysis report containing visual images, detailed parameters of each segmented region, segmentation accuracy evaluation indicators, and statistical analysis and trend prediction based on the segmentation results for industrial production management.

10. An electronic device, characterized in that, The electronic device includes: One or more processors, a memory, a graphics card, and one or more programs; Wherein, the one or more programs are stored in the memory and are configured to be executed by the one or more processors and the graphics card, and the one or more programs can be used to execute the segmentation method for uneven distribution of metal surface defect features described in any one of claims 1 to 7.

Images