A method, system and device for segmenting uneven distribution of defect features on a metal surface

Abstract

The application discloses a metal surface defect feature distribution uneven segmentation method, system and equipment, and the segmentation method is a small sample segmentation method based on a Graph Transformer; unlike a Graph Transformer based on a graph node and an edge, the Graph Transformer in the application utilizes a patch-based graph embedding technology and a global-local graph-based Transformer technology to model complex and random metal surface defect feature distribution; the application solves the complexity and randomness problems of the metal surface defect feature distribution through the Graph Transformer, provides a new solution for small sample metal surface defect segmentation actual landing, makes the segmentation performance and analysis results more suitable for task requirements, and has a good application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of metal surface defect detection, and in particular to a metal surface defect feature uneven distribution segmentation method, system and device. BACKGROUND

[0002] In recent years, due to the scarcity of metal surface defect samples in industrial application scenarios and the high cost of manual annotation, small sample segmentation schemes have received widespread attention. However, the defect features in metal surface defect samples may be unevenly distributed, meaning that these defects have no obvious regularity in terms of location, size, shape, and other features on the metal sample, and randomly appear at various locations on the sample. There may be complex relationships between different defects, such as some defects may appear together with other defects, or the presence of one defect may affect the formation and development of another defect. This randomness and complexity makes traditional small sample detection methods computationally complex and difficult to accurately identify and understand these defects. SUMMARY

[0003] To solve the above problems, the present application aims to provide a metal surface defect feature uneven distribution segmentation method, system and device.

[0004] To achieve the above technical purposes, the technical solutions of the present application are as follows:

[0005] In a first aspect, the present application provides a metal surface defect feature uneven distribution segmentation method, which is a small sample metal surface defect segmentation algorithm based on Graph Transformer. The Graph Transformer utilizes Patch-based graph embedding technology and global-local graph-based Transformer technology to model complex and random metal surface defect feature distribution.

[0006] The Patch-based graph embedding technology embeds the input features into a graph structure after dividing them into multiple equal-sized Patches through convolution, dimension reduction and linear activation operations, achieving dense prediction and linear-level calculation of defects.

[0007] The global-local graph-based Transformer technology embeds the global graph and local graph of support-query pairs into (query, key, value) to realize a self-attention mechanism.

[0008] The self-attention mechanism calculates the attention scores of different Patches through the (query, key) of the local graph, and jointly acts on the (value) of the local graph and the (query, key, value) of the global graph to realize attention on the defect topology.

[0009] As a possible implementation, further, the global graph obtains the global topology of the metal surface defect sample by embedding the support-query pair Patches into the graph structure; and the local graph obtains the local topology of the sample by embedding the masked support Patches and the query Patches into the graph structure.

[0010] The support-query pair is a task setting in a small sample meta-learning framework; and the (query, key, value) belongs to a Transformer framework and is used to calculate an attention score.

[0011] As a possible implementation, further, the meta-learning framework sets the input data as an N-way K-shot task and divides it into a support set and a query set, simulates a small amount of labeled data in a real task, and guides the model to train;

[0012] Wherein, N-way indicates that the input data contains N categories, and K-shot indicates that each category contains K images; the support set is a sample set used for model training and learning, and the query set is a sample set used for testing and evaluating the performance of the model.

[0013] As a possible implementation, further, the task setting of the small sample segmentation algorithm is 1-way K-shot, and an end-to-end training mode is adopted, and a binary cross-entropy loss and a graph structure alignment loss are set in the downstream task:

[0014]

[0015] Wherein, The binary cross-entropy loss guides the gradient update of the model by evaluating the distance between the predicted label and the real label; The graph structure alignment loss assists in guiding the gradient update of the model by evaluating the difference of the alignment operation in the process of embedding the image into the Patch and in the process of embedding the Patch into the graph structure; The is a hyperparameter.

[0016] As a possible implementation, further, the small sample metal surface defect segmentation algorithm based on the Graph Transformer includes a Patch encoder, a graph encoder, a Graph Transformer, and a residual decoder;

[0017] The Patch encoder is provided with an FPN encoder and a Patch embedding layer;

[0018] The FPN encoder encodes the 1-way K-shot support-query pair sample into one query feature and one support feature;

[0019] The Patch embedding layer divides the query feature, the support feature and the mask corresponding to the support feature into query Patches, support Patches and masked support Patches through a scale linear activation layer;

[0020] The graph encoder fuses the query Patches and the support Patches and encodes into a global graph through a graph transformation layer, and fuses the query Patches and the masked support Patches and encodes into a local graph through the graph transformation layer, and then extracts the features of the global graph and the local graph through a graph convolution network respectively;

[0021] The global graph fuses the complete support Patches and the query Patches, and can represent the global distribution of the topological structure; the local graph fuses the masked support Patches and the query Patches, and can represent the topological structure of the local defect distribution; the global graph and the local graph regard each Patches as a node, the channel feature of each Patches as the information contained by the node, and the relative position relationship between the Patches as the edge weight;

[0022] The Graph Transformer embeds the global graph and the local graph into a multi-head attention based on the (query, key, value) paradigm, calculates the attention score corresponding to the defect topological structure, and injects into the global graph and the local graph respectively;

[0023] The residual decoder is provided with a mapping layer and a residual decoding layer; the mapping layer maps the global graph and the local graph after the Graph Transformer from a graph space back to a feature space to obtain global features and local features; wherein the graph space is used to describe the space where the Graph Transformer is located; the feature space is used to describe the space where the Patch encoder, the graph encoder and the residual decoder are located;

[0024] The residual decoding layer fuses the global features, the local features and the query features, and decodes into the final prediction result for calculating the binary cross-entropy loss; the residual decoding layer also separately decodes the local features to obtain an intermediate prediction result for calculating the graph structure alignment loss.

[0025] As a possible implementation, further, the size of the query Patches, the support Patches and the masked support Patches is controlled by a hyperparameter, which ensures that the width and height thereof can be divided by the width and height of the support-query pair feature; the width and height of the support-query pair feature are determined by the size of the input image and the FPN encoder.

[0026] As a possible implementation, further, the binary cross-entropy loss reduces the difference between the true distribution and the predicted distribution, which can be expressed as:

[0027]

[0028] where N represents the number of samples, y represents the true distribution of the sample, and p represents the predicted distribution of the sample; the graph structure alignment loss can reduce the difference in the conversion process between the feature in the graph space and the feature in the feature space, which can be expressed as:

[0029]

[0030] where N represents the number of samples, y represents the true distribution of the sample, and p represents the predicted distribution of the sample.

[0031] In a second aspect, the present application provides a metal surface defect detection system, comprising a pipeline sample management system and a data analysis system;

[0032] The pipeline sample management system uses 5G wireless technology and high-speed industrial cameras to collect and transmit sample images in real time for processing by the data analysis system;

[0033] The data analysis system is loaded with the above-mentioned metal surface defect feature distribution uneven segmentation method for processing of input data and exporting of corresponding analysis reports;

[0034] The analysis report includes visualized images of segmentation results, detailed parameters (such as area, perimeter, shape features, etc.) of each segmentation region, segmentation accuracy evaluation indicators (such as Dice coefficient, Jaccard index, etc.), and statistical analysis and trend prediction based on segmentation results.

[0035] In a third aspect, the present application provides a metal surface defect detection method, comprising the following steps:

[0036] S101, obtaining metal surface sample images on the pipeline through high-speed industrial cameras and transmitting them to an industrial computer through 5G wireless technology;

[0037] S102, the industrial computer is loaded with the above-mentioned metal surface defect feature distribution uneven segmentation method, which segments the input defect samples in real time and transmits the segmentation results to the data analysis system;

[0038] S103, the data analysis system generates an analysis report containing visualized images, detailed parameters of each segmentation region, segmentation accuracy evaluation indicators, and statistical analysis and trend prediction based on segmentation results according to the segmentation results, which is used for industrial production management.

[0039] In a fourth aspect, the present application also provides an electronic device, comprising:

[0040] one or more processors, a memory, a graphics card, and one or more programs;

[0041] The one or more programs are stored in the memory and configured to be executed by the one or more processors and the graphics card, and the one or more programs can be used to execute the above-mentioned metal surface defect feature distribution uneven segmentation method.

[0042] Compared with the prior art, the technical scheme has the beneficial effects that:

[0043] The present application solves the complexity and randomness of metal surface defect feature distribution by using a small sample metal surface defect segmentation algorithm based on Graph Transformer, provides a new solution for the actual landing of small sample metal surface defect segmentation, and makes the segmentation performance and analysis results more suitable for task requirements.

[0044] The small sample segmentation algorithm for metal surface defect feature distribution unevenness provided by the present application realizes dense prediction and linear level calculation of metal surface defects through the processing of support-query pair samples by the Patch encoder and graph encoder suitable for the meta-learning paradigm and the reprocessing of the Graph Transformer and the residual decoder; the graph embedding mode based on Patches can model the complex and random distribution of metal surface defect features; the Graph Transformer combining global graphs and local graphs realizes the synergistic effect of global and local defect distribution, and can process the complex and random topological structure of metal surface defect features; in addition, the graph alignment loss and the residual decoder suitable for mapping global graphs and local graphs to a feature space and completing the final prediction can optimize the gradient calculation and back propagation of the model.

[0045] The present application realizes the whole process management of metal surface defect detection on an industrial pipeline by designing a metal surface defect detection system, method and an electronic device combined with a small sample segmentation algorithm, improves the detection efficiency, and greatly reduces the labor cost. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0047] Figure 1A data processing flowchart of the Patch encoder in the present application is shown in FIG. 1.

[0048] Figure 2 A data processing flowchart of the Graph encoder in the present application is shown in FIG. 2.

[0049] Figure 3 A data processing flowchart of the Graph Transformer in the present application is shown in FIG. 3.

[0050] Figure 4 A data processing flowchart of the residual decoder in the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0051] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It is particularly pointed out that the following embodiments are only used to illustrate the present application, but do not limit the scope of the present application. Similarly, the following embodiments are only part of the embodiments of the present application, not all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of the present application.

[0052] Embodiment 1

[0053] In order to alleviate the complexity and randomness of the distribution of metal surface defect features, the present embodiment provides a small sample segmentation method based on Graph Transformer, which is a small sample metal surface defect segmentation algorithm based on Graph Transformer. Unlike the existing Graph Transformer based on graph nodes and edges, the Graph Transformer in the present embodiment utilizes Patch-based graph embedding technology and global-local graph-based Transformer technology to model complex and random metal surface defect feature distribution.

[0054] In the present embodiment, the Patch-based graph embedding technology embeds the input features into the graph structure after dividing them into multiple equal-size Patches through convolution, dimension reduction and linear activation operations, achieving dense prediction and linear-level calculation of defects.

[0055] In this embodiment, the global-local graph-based Transformer technology embeds the global graph and the local graph of the support-query pair into (query, key, value) to realize the self-attention mechanism; wherein the support-query pair is a task setting in the small sample meta-learning framework, (query, key, value) belongs to the Transformer framework and is used to calculate the attention score; the global graph obtains the global topology structure of the metal surface defect sample by embedding the Patches of the support-query pair into the graph structure; and the local graph obtains the local topology structure of the sample by embedding the masked support Patches and the query Patches into the graph structure.

[0056] The self-attention mechanism calculates the attention scores of different Patches through the (query, key) of the local graph, and jointly acts on the (value) of the local graph and the (query, key, value) of the global graph to realize the attention on the defect topology structure.

[0057] In this embodiment, the meta-learning framework sets the input data as an N-way K-shot task and divides it into a support set and a query set to simulate a small amount of labeled data in a real task and guide the model to train. Wherein, N-way means that the input data contains N categories, K-shot means that each category contains K images, the support set is a sample set for model training and learning, and the query set is a sample set for testing and evaluating the performance of the model.

[0058] The same as the existing method, the task setting of the small sample segmentation algorithm in this embodiment is 1-way K-shot, and an end-to-end training method is adopted; the difference lies in that a binary cross-entropy loss and a graph structure alignment loss are set in the downstream task in this embodiment:

[0059]

[0060] In the above formula, is a binary cross-entropy loss, is a graph structure alignment loss, is a hyperparameter. Wherein, the binary cross-entropy loss evaluates the distance between the predicted label and the real label to guide the gradient update of the model; and the graph structure alignment loss evaluates the difference of the alignment operation in the process of embedding the image into the Patch and in the process of embedding the Patch into the graph structure to assist in guiding the gradient update of the model.

[0061] In this embodiment, the small sample metal surface defect segmentation algorithm based on the Graph Transformer includes a Patch encoder 100, a graph encoder 110, a Graph Transformer 120 and a residual decoder 130.

[0062] Referring to the accompanying drawings Figure 1As shown, the Patch encoder 100 is provided with an FPN encoder 101 and a Patch embedding layer 102; the FPN encoder 101 encodes a 1-way K-shot support-query pair sample into 1 query feature 103 and 1 support feature 104, and the Patch embedding layer 102 utilizes the query feature 103, the support feature 104 and a mask 105 corresponding to the support feature, and divides them into query Patches 107, support Patches 108 and masked support Patches 109 through a scale linear activation layer 106, the size of the query Patches 107, the support Patches 108 and the masked support Patches 109 is controlled by a hyperparameter, ensuring that the width and height thereof can be evenly divided by the width and height of the support-query pair feature, and the width and height of the support-query pair feature are determined by the size of the input image and the FPN encoder 101.

[0063] Referring to the accompanying drawings Figure 2 As shown, the graph encoder 110 fuses the query Patches 107 and the support Patches 108 and encodes them into a global graph 112 through a graph transformation layer 111; at the same time, the graph encoder 110 fuses the query Patches 107 and the masked support Patches 109 and encodes them into a local graph 113 through the graph transformation layer 111; then the features of the global graph 112 and the local graph 113 are extracted through a graph convolution network respectively. Among them, the global graph 112 fuses the complete support Patches 107 and the query Patches 107, and can represent the topological structure of the global distribution; the local graph 113 fuses the masked support Patches 109 and the query Patches 107, and can represent the topological structure of the local defect distribution; the global graph 112 and the local graph 113 regard each Patch as a node, the channel feature of each Patch as the information contained by the node, and the relative position relationship between the Patches as the edge weight.

[0064] Referring to the accompanying drawings Figure 3 As shown, the Graph Transformer 120 embeds the global graph 112 and the local graph 113 into a multi-head attention 121 based on the (query, key, value) paradigm, calculates the attention score 122 corresponding to the defect topological structure, and injects it into the global graph 112 and the local graph 113 respectively.

[0065] Referring to the accompanying drawings Figure 4As shown, the residual decoder 130 is provided with a mapping layer 131 and a residual decoding layer 132; the mapping layer 131 maps the global graph 112 and the local graph 113 from the graph space 133 back to the feature space 134 to obtain the global feature 135 and the local feature 136. The graph space 133 is used to describe the space where the Graph Transformer 120 is located; the feature space 134 is used to describe the space where the Patch encoder 100, the graph encoder 110 and the residual decoder 130 are located.

[0066] The residual decoding layer 132 fuses the global feature 135, the local feature 136 and the query feature 103, and decodes into the final prediction result 137 for calculating the binary cross-entropy loss; the residual decoding layer 132 also decodes the local feature 136 alone to obtain the intermediate prediction result 138 for calculating the graph structure alignment loss.

[0067] The binary cross-entropy loss reduces the difference between the real distribution and the predicted distribution, and can be expressed as:

[0068]

[0069] In the above formula, N represents the number of samples, y represents the real distribution of the sample, and p represents the predicted distribution of the sample. The graph structure alignment loss can reduce the difference in the conversion process between the feature space and the feature space, and can be expressed as:

[0070]

[0071] In the above formula, N represents the number of samples, y represents the real distribution of the sample, and p represents the predicted distribution of the sample.

[0072] Embodiment 2

[0073] The embodiment provides a metal surface defect detection system, which comprises a pipeline sample management system and a data analysis system.

[0074] The pipeline sample management system uses 5G wireless technology and high-speed industrial cameras to collect and transmit sample images in real time for processing by the data analysis system.

[0075] The data analysis system is loaded with the metal surface defect feature distribution uneven segmentation method in embodiment 1 for fast processing of input data and exporting corresponding analysis reports; the analysis reports include visual images of segmentation results, detailed parameters (such as area, perimeter, shape features, etc.) of each segmentation region, segmentation accuracy evaluation indexes (such as Dice coefficient, Jaccard index, etc.), as well as statistical analysis and trend prediction based on the segmentation results.

[0076] Embodiment 3

[0077] The embodiment provides a metal surface defect detection method, comprising the following steps:

[0078] In S101, a pipeline sample management system acquires metal surface sample images on a pipeline through a high-speed industrial camera and transmits the images to an industrial computer through 5G wireless technology.

[0079] In S102, the industrial computer is loaded with the metal surface defect feature uneven distribution segmentation method in the embodiment 1, and real-time segmentation is performed on the input defect sample, and the segmentation result is transmitted to a data analysis system.

[0080] In S103, the data analysis system generates an analysis report containing a visual image, detailed parameters of each segmentation region, segmentation precision evaluation indexes, statistical analysis and trend prediction based on the segmentation result, and the analysis report is used for industrial production management.

[0081] Embodiment 4

[0082] Based on the same inventive concept, the embodiment provides an electronic device, which comprises:

[0083] one or more processors, a memory, a graphics card, and one or more programs;

[0084] The one or more programs are stored in the memory and configured to be executed by the one or more processors and the graphics card, and the one or more programs can be used to execute the metal surface defect feature uneven distribution segmentation method in the embodiment 1.

[0085] The above only describes some embodiments of the present application, and does not limit the protection scope of the present application, and any equivalent device or equivalent process transformation obtained by using the content of the present application specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for segmenting unevenly distributed defect features on a metal surface, characterized in that, The method is a small sample metal surface defect segmentation algorithm based on a GraphTransformer, wherein the Graph Transformer utilizes a Patch-based graph embedding technology and a global-local graph-based Transformer technology to model complex and random metal surface defect feature distribution; The Patch-based graph embedding technology embeds input features into a graph structure after dividing the input features into multiple equal-size Patches through convolution, dimension reduction and linear activation operations; The global-local graph-based Transformer technology embeds global graphs and local graphs of support-query pairs into queries, keys and values to realize a self-attention mechanism; wherein the support-query pairs are task settings in a small sample meta-learning framework; the queries, keys and values belong to a Transformer framework and are used to calculate attention scores The self-attention mechanism calculates attention scores of different Patches through queries and keys of the local graphs, and jointly acts on values of the local graphs and queries, keys and values of the global graphs to realize attention on defect topological structures; The small sample metal surface defect segmentation algorithm based on the GraphTransformer comprises a Patch encoder, a graph encoder, a Graph Transformer and a residual decoder; The FPN encoder encodes a 1-way K-shot support-query pair sample into one query feature and one support feature; The Patch embedding layer divides the query features, the support features and the support feature masks into equal-size query Patches, support Patches and masked support Patches through a scale linear activation layer; the sizes of the query Patches, the support Patches and the masked support Patches are controlled by a hyperparameter, so that the width and height of the Patches can be evenly divided by the width and height of the support-query pair features; the width and height of the support-query pair features are determined by the size of an input image and the FPN encoder; The graph encoder fuses the query Patches and the support Patches and encodes them into a global graph through a graph transformation layer, and simultaneously fuses the query Patches and the masked support Patches and encodes them into a local graph through the graph transformation layer, and then extracts features of the global graph and the local graph through a graph convolution network; The global graph and the local graph regard each Patches as a node, the channel features of each Patches as information contained in the node, and the relative position relationship between the Patches as an edge weight; The Graph Transformer embeds the global graph and the local graph into a multi-head attention based on a query, key and value paradigm, calculates attention scores corresponding to defect topological structures, and injects the attention scores into the global graph and the local graph, respectively. The residual decoder is provided with a mapping layer and a residual decoding layer; the mapping layer maps the global graph and the local graph after the Graph Transformer from a graph space back to a feature space to obtain global features and local features; wherein the graph space is used to describe the space where the Graph Transformer is located; the feature space is used to describe the space where the Patch encoder, the graph encoder and the residual decoder are located; The residual decoding layer fuses the global features, the local features and the query features, and decodes them into final prediction results for calculating a binary cross-entropy loss; the residual decoding layer also separately decodes the local features to obtain intermediate prediction results for calculating a graph structure alignment loss.

2. The metal surface defect feature distribution uneven segmentation method according to claim 1, characterized in that, The global graph obtains the global topological structure of the metal surface defect sample by embedding the Patches of the support-query pair into a graph structure; and the local graph obtains the local topological structure of the sample by embedding the masked support Patches and the query Patches into a graph structure.

3. The metal surface defect feature distribution uneven segmentation method according to claim 2, characterized in that, The meta-learning framework sets the input data as an N-way K-shot task, and divides it into a support set and a query set, simulates a small amount of labeled data in a real task, and guides the model to train; Wherein, N-way represents that the input data contains N categories, and K-shot means that each category contains K images; the support set is a sample set for model training and learning, and the query set is a sample set for testing and evaluating the performance of the model.

4. The metal surface defect feature distribution uneven segmentation method according to claim 3, characterized in that, The task setting of the small sample segmentation algorithm is 1-way K-shot, and an end-to-end training mode is adopted, and a binary cross-entropy loss and a graph structure alignment loss are set in the downstream task: wherein, is a binary cross-entropy loss, the binary cross-entropy loss guides the gradient update of the model by evaluating the distance between the predicted label and the true label; is a graph structure alignment loss, the graph structure alignment loss assists in guiding the gradient update of the model by evaluating the difference of the alignment operation in the process of embedding the image into the patch and in the process of embedding the patch into the graph structure; is a hyperparameter.

5. The metal surface defect feature distribution uneven segmentation method of claim 1, wherein, The binary cross-entropy loss reduces the difference between the real distribution and the predicted distribution, and is expressed as: Wherein, N represents the number of samples, y represents the real distribution of the sample, and p represents the predicted distribution of the sample; the graph structure alignment loss can reduce the difference in the conversion process between the feature in the graph space and the feature in the feature space, and is expressed as: Wherein, N represents the number of samples, y represents the real distribution of the sample, and p represents the predicted distribution of the sample.

6. A metal surface defect detection system characterized by, It comprises a pipeline sample management system and a data analysis system; The pipeline sample management system uses 5G wireless technology and a high-speed industrial camera to collect and transmit sample images in real time for processing by the data analysis system; The data analysis system is loaded with the metal surface defect feature distribution uneven segmentation method according to any one of claims 1-5, for processing of input data, and exporting a corresponding analysis report; The analysis report contains a visual image of the segmentation result, detailed parameters of each segmentation area, segmentation accuracy evaluation indexes, and statistical analysis and trend prediction based on the segmentation result.

7. A method of detecting defects on a metal surface, characterized by, It comprises the following steps: S101, acquire the metal surface sample image on the pipeline through a high-speed industrial camera, and transmit it to an industrial computer through 5G wireless technology; S102, the industrial computer is loaded with the metal surface defect feature distribution uneven segmentation method according to any one of claims 1-5, which segments the input defect sample in real time and transmits the segmentation result to the data analysis system; S103, the data analysis system generates an analysis report containing a visualization image, detailed parameters of each segmented region, segmentation accuracy evaluation indexes, and statistical analysis and trend prediction based on the segmentation result, for industrial production management.

8. An electronic device, comprising: The electronic device includes: one or more processors, a memory, a graphics card, and one or more programs; wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors and the graphics card, and the one or more programs can be used to perform the segmentation method for uneven distribution of metal surface defect features as claimed in any one of claims 1-5.

Images