Strip defect segmentation method based on Markov reconstruction and correlation space alignment

Abstract

The present invention provides a strip defect segmentation method based on Markov reconstruction and association space alignment, which relates to the technical field of strip surface defect detection. The method comprises: collecting original images of the strip surface, including defect-free images and defect images of different categories; annotating the collected images and dividing the data sets; constructing input data groups in pre-training tasks and downstream tasks respectively; extracting features from the input data groups to obtain embedded feature vectors of the input strip surface defect semantic segmentation base model; constructing the strip surface defect semantic segmentation base model for strip surface defect segmentation. The strip surface defect semantic segmentation base model comprises: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module and an association space alignment module. The present invention can achieve high-precision semantic segmentation for defect samples that have never been trained in a completely new scenario.

Description

Technical Field

[0001] The present invention relates to the technical field of strip steel surface defect detection, and in particular to a strip steel defect segmentation method based on Markov reconstruction and correlation space alignment. Background Art

[0002] Strip steel is an essential industrial material, widely used in a variety of fields, including automotive, shipbuilding, and aerospace. However, during the hot rolling process, due to production process and equipment limitations, defects such as cracks, scratches, and patches often appear on the strip's surface. These surface defects not only affect the steel's wear resistance, corrosion resistance, and fatigue strength, but also seriously impact the overall quality and service life of the product. Therefore, to ensure safe use and avoid potential risks, rigorous surface inspection is essential during the strip production process to promptly detect and address these issues.

[0003] Manual visual inspection for defects in high-speed moving steel strips is plagued by high workload, low efficiency, and high rates of missed and false detections. Traditional machine vision-based inspection technology, due to limitations such as poor generalization and difficulty constructing features, has been replaced by AI-based deep learning. High-resolution cameras capture images of the strip surface and use semantic segmentation methods to perform high-precision pixel-level segmentation, locating defect areas and enabling fine-grained defect recognition.

[0004] However, current semantic segmentation methods have the following problems:

[0005] (1) The existing pre-training mechanism of the strip surface image base model is based on fine-grained texture learning or representation space alignment. These two training paradigms can only constrain parameter learning from one perspective and cannot fully explore the background patterns and defect knowledge of the strip images during the pre-training process. Considering that strip surface images of the same batch have similar backgrounds but variable defects, as the base model for the downstream high-precision fine-grained semantic segmentation task, its limitations restrict further improvement of the subtask accuracy.

[0006] (2) The current small-sample downstream training paradigm is mainly based on the construction of adapters to adjust parameter weights, which cannot fully exploit the characteristics of new scene samples for feature alignment. Compared with the natural image field, industrial scenes present challenges of new categories and complex samples. The current methods have limited improvement in the fine-grained segmentation performance of new scene and new defect samples.

[0007] (3) Existing target sample information extraction strategies are unable to capture the key features of the defective parts of the image to be identified, and do not fully utilize the reference new scene samples to extract associations between the images to be identified. Existing feature matching methods feed the segmentation results of new scene samples into the network as mask features for learning, but are unable to use the key information of the category to guide the extraction of visual features, resulting in poor segmentation performance.

[0008] (4) Existing downstream fine-tuning methods require training additional decoder heads based on new defect categories. The model then segments the image to be identified by matching it with the learned model parameters. However, due to the differences in the morphology of new defect categories and the lack of a large amount of fine-tuning data in actual industrial conditions, the model's generalization ability and segmentation quality are affected.

[0009] (5) Finally, considering the strict product quality requirements of strip steel production, it is difficult to collect sufficient strip steel surface defect data in a completely new environment. The sparse training data makes it easy for the semantic segmentation model to overfit when fine-tuning the training in the new scenario. In addition, for defect forms that have never appeared before, the model lacks the features and knowledge learning of large-scale training data, making it difficult to effectively extract image information for high-precision segmentation.

[0010] Therefore, there is an urgent need for a strip surface defect segmentation method that can meet the requirements of high-precision semantic segmentation without training in new scenarios. Summary of the Invention

[0011] In response to the above problems, the purpose of the present invention is to provide a strip defect segmentation method based on Markov reconstruction and correlation space alignment, which can achieve high-precision semantic segmentation for defect samples that have never been trained in a new scenario.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0013] In one aspect, a strip defect segmentation method based on Markov reconstruction and correlation space alignment is provided, the method comprising the following steps:

[0014] S1. Collect original images of the strip surface, including defect-free images and images of different types of defects;

[0015] S2. Label the defect targets in the defect image, perform one-hot encoding on all defect images according to different categories, and treat each defect image and the corresponding one-hot encoding label as a defect data pair;

[0016] S3. All defect-free images are used as pre-training task datasets, and all defect data pairs are divided into downstream task training sets and downstream task test sets in proportion as the complete downstream task dataset;

[0017] S4. Construct input data sets for pre-training tasks and downstream tasks respectively;

[0018] In the pre-training task, each input data set includes a defect-free image and the corresponding text description; in the downstream task, each input data set includes the target defect image and target defect mask, the reference defect image and reference defect mask, the iteration time step and the category text data;

[0019] S5. Perform feature extraction on the input data group to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model;

[0020] S6. Constructing a strip steel surface defect semantic segmentation base model for strip steel surface defect segmentation;

[0021] The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module;

[0022] The obtained embedded feature vector is sent to two weight-shared Markov noise transfer matrix reconstruction networks, and the visual feature constraint module and the detail information constraint module are used to perform targeted constraints on the network training process from the two aspects of visual features and detail information; the association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, matches the association areas on the target defect image and the reference defect image, and obtains the final prediction result.

[0023] Optionally, in step S2, all defect images are one-hot encoded according to different categories, including nine defect categories: inclusions, scratches, holes, red iron sheets, iron sheet gray, plate oxide scale, roller marks, water stains and oil stains; the pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background, marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

[0024] Optionally, in step S3, all defect-free images are used as pre-training task data sets, and the text description of each defect-free image is ; at the same time, all defect data pairs are divided into a downstream task training set and a downstream task test set in a ratio of 8:2 as a complete downstream task data set.

[0025] Optionally, in the step S4, in the pre-training task, each input data group includes a defect-free image and a corresponding text description, which are used to pre-train the strip surface defect semantic segmentation base model;

[0026] In the downstream task, each input data set includes the target defect image IT , target defect mask M T , reference defect image I S , reference defect mask M S , iterate time step t and category text data T;

[0027] Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect dataset R∈N, where N is the number of all defect data pairs of this category. Excluding the current defect image, a defect data pair is randomly selected from the subset R'∈N-1 of the optional defect dataset to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

[0028] Optionally, in step S5, feature extraction is performed on the input data set, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector;

[0029] The step S5 specifically includes:

[0030] For the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is fed into the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iterative time step t t ;

[0031] The iterative time step t is passed through the linear embedding layer to increase the dimension and obtain the time step embedding vector y t , used for time step indication in feature space; target defect image I T After comparing with the reference defect image IS After the same data preprocessing operation, they are respectively sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features at different levels are collected, and after noise construction at the same iterative time step t, the target defect noise feature x that conforms to the Gaussian distribution is obtained. t ; Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t ;

[0032] The target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain fine-grained texture detail information at the image level; the category text data T representing the target defect mask category is passed through the text feature encoder E T Converted into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , perform targeted sequence information conversion according to different texts to obtain serialized text vectors.

[0033] Optionally, the method further includes: using the pre-training task data set to pre-train the Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

[0034] Optionally, the Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition coding layer and three up-sampling sequence feature decoding modules;

[0035] The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature, both of which have the same spatial dimension. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t ;

[0036] The transition coding layer further distinguishes the regional information of the visual feature prediction sequence by adjusting the feature dimension, including defect part features and background general features;

[0037] The three up-sampled sequence feature decoding modules are used to perform cubic interpolation up-sampling on the visual feature prediction sequence and perform Gaussian noise constraint; the self-attention layer output features in the three up-sampled sequence feature decoding modules are combined into a reference defect image visual feature sequence Or target defect image visual feature sequence The output sequences of the three up-sampling sequence feature decoding modules are used as detail information constraint sequences n t ;

[0038] The serialized text vector and time step embedding vector are fed into the cross-fusion layer of each scale module of the network for fusion and embedding, respectively realizing text guidance and time step specification.

[0039] Optionally, in the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment, specifically including: visual feature prediction sequence m t After being mapped to the visual feature constraint sequence f by a multi-layer perceptron t The same dimensional space is then used to calculate the negative cosine similarity, and the constraint control is performed using the pre-set adjustable constraint parameter λ.

[0040] Optionally, in the detail information constraint module, a linear transformation layer is used to constrain the acquired detail information sequence n t Perform dimensionality reduction, and then use matrix dimension transformation to convert one-dimensional data into two-dimensional data; obtain the defect mask for detail information constraint by interpolation upsampling and sequence splicing, normalize the channels by taking the mean, and compare it with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

[0041] Optionally, in the association space alignment module, the reference defect image visual feature sequence and the target defect image visual feature sequence Perform correlation alignment in the sequence correlation space, specifically including:

[0042] The visual feature sequence of the reference defect image After the linear layer is mapped to the visual feature sequence of the target defect image The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication, and the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication is performed to obtain the target defect feature portion X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimension is transformed to obtain X' map ; After passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the predicted defect mask M is obtained P As the final prediction output, and in the training stage by comparing it with the target defect mask M T Calculate the binary cross entropy as the loss function for constraint.

[0043] In another aspect, an electronic device is provided, comprising:

[0044] processor;

[0045] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the strip defect segmentation method based on Markov reconstruction and association space alignment are implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which program code is stored. The program code can be called by a processor to execute the steps of the above-mentioned strip defect segmentation method based on Markov reconstruction and correlation space alignment.

[0047] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0048] (1) Based on the Markov reconstruction training method, the present invention provides a powerful segmentation model pre-training constraint, which can decode and reconstruct the pixels of the entire image, analyze the detailed information, and through feature prediction and alignment constraints, enable the model to focus on capturing high-frequency information and key information in the image, further enhancing the base model's ability to understand the visual features of the strip surface image.

[0049] (2) The association space alignment module based on matrix calculation in the present invention can perform association calculation on the visual features of the specified layer and use a strict training task constraint model to mine the subtle associations between features, thereby realizing the prediction of similar defect areas between new defect samples and images to be segmented, thereby improving the model's accurate segmentation capability.

[0050] (3) Based on the matching pattern of reference defect images, this invention enables the model to achieve accurate segmentation of the target defect to be predicted in a new scenario without training, and automatically matches the relevant areas on the image. Given a defect text category, similar features are matched only by a reference sample and its mask, which enhances the practicality and generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a schematic diagram of a strip defect segmentation method based on Markov reconstruction and correlation space alignment provided by an embodiment of the present invention;

[0053] Figure 2Schematic diagram of a base model for feature extraction and inputting semantic segmentation of steel strip surface defects provided by an embodiment of the present invention;

[0054] Figure 3 Schematic diagram of a Markov noise transfer matrix reconstruction network provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of a visual feature constraint module provided by an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a detail information constraint module provided by an embodiment of the present invention;

[0057] Figure 6 4 is a schematic diagram of an associative space alignment module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" in the present invention should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner.

[0060] The embodiment of the present invention provides a strip defect segmentation method based on Markov reconstruction and correlation space alignment, such as Figure 1 As shown, the overall process of this method is as follows:

[0061] In the semantic segmentation scenario of strip steel surface defects, a visible light camera is used to capture RGB strip steel surface images. After fine-grained and high-precision annotation, the dataset is divided proportionally. The obtained strip steel surface defect dataset, for each target defect image that needs to be segmented in the current scenario, has an input data set including a reference defect image and its corresponding mask for segmentation reference, a target defect image and its corresponding mask, text data indicating the defect category of the current target, and the iterative time step of the Markov chain noise. The above data is fed into a combined data feature extraction and embedding network. A specified encoder is used to extract features from the image, text, and time step, encoding them into an embedded feature vector that can be fused and aligned. Subsequently, the obtained embedded feature vector is fed into the base model for semantic segmentation of strip steel surface defects.

[0062] The base model for semantic segmentation of strip surface defects includes two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module. In the Markov noise transfer matrix reconstruction network, texture information is extracted through convolution operations, self-attention operations focus on important semantic parts of the image, and a cross-fusion layer aligns text and visual information, focusing on defect area information in the multi-scale receptive field of the image to obtain fine-grained texture features of the specified image. The network uses four loss functions to strictly constrain the network's ability to mine information and the effectiveness of vector features for high-precision segmentation. Gaussian noise prediction is used to indirectly implement the image reconstruction paradigm, and resampling reduces the difficulty of model fitting and improves convergence speed.

[0063] Furthermore, the training task of the base model is constrained by the visual feature constraint module and the detail information constraint module, respectively, from the perspectives of image texture information and feature sequence space, improving the network's generalization in downstream defect recognition. Finally, a correlation space alignment module is designed to align the visual feature sequences of the reference defect and the target defect in the sequence correlation space, obtaining defect regions on the target defect image that are similar to the reference defect image. The cross-entropy loss function is then used to constrain the model to learn the association between the visual feature sequences of the two images, thereby achieving high-precision semantic segmentation without training in new downstream scenarios and new defects.

[0064] The method specifically comprises the following steps:

[0065] Step 1: Collect original images of the strip surface, including defect-free images and images with different types of defects.

[0066] In this step, the original images of the strip surface are captured by an RGB camera, and 12,000 defect-free images and 4,000 defect images of different categories are collected.

[0067] Step 2: Label the defect targets in the defect image, perform one-hot encoding on all defect images according to different categories, and treat each defect image and the corresponding one-hot encoding label as a defect data pair.

[0068] In this step, all defect images are one-hot encoded according to different categories, including inclusions, scratches, holes, red iron sheets, iron sheet dust, plate oxide scale, roller marks, water stains and oil stains. The pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background, marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

[0069] Step 3: All defect-free images are used as pre-training task datasets. At the same time, all defect data pairs are divided into downstream task training set and downstream task test set in proportion as the complete downstream task dataset.

[0070] In this step, all defect-free images are used as the pre-training dataset, with each image described as "a defect-free image of a steel strip surface." Furthermore, all defect data pairs are divided into a downstream task training set and a downstream task test set in an 8:2 ratio, forming the complete downstream task dataset. The model learns the associations of the defects to be identified using the downstream task training set, and then verifies the learned model's segmentation performance under novel defects and novel scenarios using the downstream task test set.

[0071] Step 4: Construct input data groups in pre-training tasks and downstream tasks respectively.

[0072] In the pre-training task, each input data set includes a defect-free image and its corresponding text description, which is used to pre-train the base model for semantic segmentation of strip surface defects. By feeding a large number of defect-free strip surface images, the model deeply mines image texture information in the Markov reconstruction task, learns to fit similar image distributions, and acquires the ability to extract visual features from downstream strip surface defect images.

[0073] In the downstream task, an input data set is constructed for each original defect data pair. Each input data set includes the target defect image I T , target defect mask M T , reference defect image I S , reference defect mask M S , iterate time step t and category text data T;

[0074] Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect dataset R∈N, where N is the number of all defect data pairs of this category. Excluding the current defect image, a defect data pair is randomly selected from the subset R'∈N-1 of the optional defect dataset to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

[0075] Step 5: Extract features from the input data set to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model.

[0076] In this step, feature extraction is performed on the input data group, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector.

[0077] like Figure 2 As shown, for the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is fed into the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iterative time step t t .

[0078] At the same time, the iterative time step t is upgraded through the linear embedding layer to obtain the time step embedding vector y t , used for time step indication in feature space. Target defect image I T After comparing with the reference defect image I S After the same data preprocessing operation, they are respectively sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features at different levels are collected, and after noise construction at the same iterative time step t, the target defect noise feature x that conforms to the Gaussian distribution is obtained. t . Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t .

[0079] At the same time, the target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain the fine-grained texture detail information at the image level. In addition, the category text data T representing the target defect mask category (such as <an image of a steel strip surface with a scratch defect>) is passed through the text feature encoder E TConverted into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , according to different texts, targeted sequence information conversion is performed to obtain a serialized text vector with a dimension of 1024.

[0080] Step 6. Construct a semantic segmentation base model for strip surface defects for strip surface defect segmentation.

[0081] The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module and an associated space alignment module.

[0082] The obtained embedded feature vector is fed into two weight-shared Markov noise transfer matrix reconstruction networks. The visual feature constraint module and the detail information constraint module are used to impose targeted constraints on the network training process from the perspectives of visual features and detail information. The association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, matching the associated regions (i.e., the same defect parts) on the target defect image and the reference defect image. This region is the final prediction result. The target defect mask is used as the pixel-level constraint of this result, and the binary cross entropy function (BCE) is used for measurement.

[0083] Step 7: Use the pre-training task dataset to pre-train the Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

[0084] The present invention conducts large-scale pre-training of the base model based on the Markov reconstruction paradigm, fully improving the base model's ability to extract visual features from the strip surface image, and feeding the text information of the defect-free image into the guidance, enhancing the model's ability to align text with the image, so that the text sequence can better implement semantic guidance for different defect categories in the downstream. The present invention constructs a Markov chain to model the image distribution, and realizes image reconstruction based on the probabilistic graphical model by gradually adding and removing the noise transfer matrix on the original image. The total time step T is set to 1000, considering the transfer from the original image x0 to the state x close to the Gaussian distribution t It can be obtained by sampling through formula (1). Therefore, a parameterized reverse Markov chain is designed to gradually transfer the standard Gaussian distribution to the original image by gradually removing the noise, that is, reconstructing the original image. In order to obtain a reverse transfer matrix that is close to the distribution of the forward noise transfer matrix, the Markov noise transfer matrix reconstruction network D is trained for prediction. The noise transfer matrix ε predicted by the network is D (x t,t) and the noise transfer matrix ε added at the corresponding time step are constrained by the mean square error (MSE).

[0085]

[0086] Where x0 is the original image, x t is the image after adding Gaussian noise.

[0087]

[0088] α i is the noise scheduling parameter, in

[0089] Step 8, such as Figure 3 As shown, the Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition coding layer and three up-sampling sequence feature decoding modules.

[0090] The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature. The spatial dimensions of the two are the same, and two identical networks with shared weights can be used. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t . Visual feature prediction sequence m t There are three dimensions (three channels), which can be written as Where i = 1, 2, 3, and the three dimensions are [32*32*256], [16*16*512], and [8*8*1024] respectively.

[0091] The transition coding layer further distinguishes the regional information of the visual feature prediction sequence by adjusting the feature dimension, including the defect part features and the background general features. The three upsampling sequence feature decoding modules with the same number of channels and dimension information as the downsampling sequence feature encoding module are used to perform cubic interpolation upsampling on the visual feature prediction sequence and perform Gaussian noise constraint. Predicting Gaussian noise Label with Gaussian noise Constraint is performed by Mean Square Error (MSE).

[0092] At the same time, the visual sequences of different nodes in the upsampling decoding process are taken out and constrained and further processed respectively to improve the accuracy and generalization of the network in predicting segmentation masks under new defects. The output features of the self-attention layer in the three upsampling sequence feature decoding modules are combined into the visual feature sequence of the reference defect image. Or target defect image visual feature sequence and There are three dimensions, which can be written as Where i = 1, 2, 3, the dimensions are [64*1280], [256*1280] and [1024*1280] respectively. The output sequences of the three up-sampled sequence feature decoding modules are used as the detail information constraint sequence n t , n t There are three dimensions, which can be written as Where i = 1, 2, 3, the dimensions are [8*8*1024], [16*16*512] and [32*32*256] respectively.

[0093] The serialized text vector and time step embedding vector are fed into the cross-fusion layer of each scale module of the network for fusion and embedding, respectively realizing text guidance and time step specification.

[0094] Step 9, such as Figure 4 As shown, in the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment. The visual feature prediction sequence and the visual feature constraint sequence have three dimensions, which can be recorded as and Where i=1,2,3.

[0095] Specifically, the visual feature prediction sequence m t After being mapped to the visual feature constraint sequence f by a multi-layer perceptron t The same dimensional space is then used to calculate the negative cosine similarity and use the pre-set adjustable constraint parameter λ for constraint control. The loss function L similarity The measurement is shown in formula (2).

[0096]

[0097] Among them, λ is an adjustable constraint parameter, and its default value is 0.5.

[0098] Step 10, such as Figure 5 As shown, in the detail information constraint module, the acquired detail information constraint sequence n t It consists of features of three different dimensions. First, the linear transformation layer is used to constrain the sequence n of detail information. tDimensionality reduction is performed to [64*1], [256*1], and [1024*1]. Matrix dimension transformation is then used to convert the one-dimensional data into two-dimensional data to restore the image spatial information. Subsequently, all image visual features are interpolated and upsampled to 64*64, and multiple sequences are spliced ​​to obtain a defect mask of dimension [64*64*3] for detail information constraint. Finally, the channel is normalized to [64*64*1] by taking the mean, and compared with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

[0099] Step 11, such as Figure 6 As shown, in the association space alignment module, the reference defect image visual feature sequence and the target defect image visual feature sequence Perform correlation alignment in the sequence correlation space and align the two visual feature sequences at the defect part through matrix calculation of the same dimension. Specifically, it includes:

[0100] The reference defect image visual feature sequence After the linear layer is mapped to the visual feature sequence of the target defect image The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication; the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication is performed to obtain the target defect feature portion X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimension is transformed to obtain X' map , X′ map =reshape(mean(concat[X map i ])), i = 1, 2, 3; After passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the texture information of the currently obtained aligned feature map can be further enhanced to obtain the predicted defect mask M P As the final prediction output, and in the training stage by comparing it with the target defect mask M T Calculate the binary cross entropy as the loss function for constraint.

[0101] In an embodiment of the present invention, in order to address the problem that existing segmentation model pre-training strategies are difficult to constrain the semantic segmentation of strip steel images at both the visual feature level and the detail information level, a training paradigm based on Markov reconstruction is designed, and a visual feature constraint module and a detail information constraint module are designed based on the output features of different network modules, so as to realize the learning of fine-grained pixel information and feature associations of the entire image, and improve the feature extraction capability of the base model.

[0102] To address the problem that existing target sample information extraction strategies are unable to capture the key features of the parts that need to be segmented in the image to be identified, the present invention proposes an association space alignment module for image visual feature sequences, which explores the association information between new reference defect samples and the image to be segmented in the feature association space, and uses the text information of the defect category to achieve prior guidance of the reference defect image, further improving the high-precision segmentation capability of new defects.

[0103] To address the problem that existing base models, after pre-training with large-scale data and large-volume parameter structures, still require a certain scale of training samples and resource requirements for fine-tuning in the face of new scenarios and new defects, the present invention designs a downstream prediction system that uses reference defect samples to perform image-specified defect recognition and segmentation. In a completely new scenario, it relies on feature pairing to segment the defect area, thereby achieving accurate segmentation of new samples without training.

[0104] In an exemplary embodiment, the present invention further provides an electronic device, comprising:

[0105] processor;

[0106] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the strip defect segmentation method based on Markov reconstruction and association space alignment are implemented.

[0107] In an exemplary embodiment, the present invention further provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the steps of the aforementioned method for strip defect segmentation based on Markov reconstruction and correlation space alignment. For example, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device.

[0108] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements but also other elements not explicitly listed, or also includes elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0109] References in the specification to "one embodiment," "an embodiment," "an exemplary embodiment," "some embodiments," etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include that particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it is within the knowledge of persons skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0110] It should be understood that the term "and / or" as used herein simply describes an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the related objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0111] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0112] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0117] The present invention encompasses any alternatives, modifications, equivalents, and solutions that fall within the spirit and scope of the present invention. To provide a thorough understanding of the present invention, specific details are described in detail below in connection with the preferred embodiments of the present invention, but those skilled in the art will be able to fully understand the present invention without these detailed descriptions. Furthermore, to avoid unnecessary confusion regarding the essence of the present invention, well-known methods, processes, procedures, components, and circuits have not been described in detail.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A strip defect segmentation method based on Markov reconstruction and correlation space alignment, characterized in that: The following steps are involved: S1. Collect original images of the strip surface, including defect-free images and images of different types of defects; S2. Label the defect targets in the defect image, perform one-hot encoding on all defect images according to different categories, and treat each defect image and the corresponding one-hot encoding label as a defect data pair; S3. All defect-free images are used as pre-training task datasets, and all defect data pairs are divided into downstream task training sets and downstream task test sets in proportion as the complete downstream task dataset; S4. Construct input data sets for pre-training tasks and downstream tasks respectively; In the pre-training task, each input data set includes a defect-free image and the corresponding text description; in the downstream task, each input data set includes the target defect image and target defect mask, the reference defect image and reference defect mask, the iteration time step and the category text data; S5. Perform feature extraction on the input data group to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model; S6. Constructing a strip steel surface defect semantic segmentation base model for strip steel surface defect segmentation; The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module; The obtained embedded feature vector is sent to two weight-shared Markov noise transfer matrix reconstruction networks, and the visual feature constraint module and the detail information constraint module are used to perform targeted constraints on the network training process from the two aspects of visual features and detail information; the association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, matches the association areas on the target defect image and the reference defect image, and obtains the final prediction result.

2. The method according to claim 1, characterized in that In step S2, all defect images are one-hot encoded according to different categories, including nine defect categories: inclusions, scratches, holes, red iron sheets, iron sheet dust, plate oxide scale, roller marks, water stains and oil stains; the pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background, marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

3. The method according to claim 1, characterized in that In step S3, all defect-free images are used as pre-training task datasets, and the text description of each defect-free image is ; at the same time, all defect data pairs are divided into a downstream task training set and a downstream task test set in a ratio of 8:2 as a complete downstream task dataset.

4. The method according to claim 1, wherein In the step S4, in the pre-training task, each input data group includes a defect-free image and a corresponding text description, which are used to pre-train the strip surface defect semantic segmentation base model; In the downstream task, each input data set includes the target defect image I T , target defect mask M T , reference defect image I S , reference defect mask M S , iterate time step t and category text data T; Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect dataset R∈N, where N is the number of all defect data pairs of this category. Excluding the current defect image, a defect data pair is randomly selected from the subset R'∈N-1 of the optional defect dataset to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

5. The method according to claim 1, wherein In step S5, feature extraction is performed on the input data set, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector; The step S5 specifically includes: For the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is fed into the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iterative time step t t ; The iterative time step t is passed through the linear embedding layer to increase the dimension and obtain the time step embedding vector y t , used for time step indication in feature space; target defect image I T After comparing with the reference defect image I S After the same data preprocessing operation, they are respectively sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features at different levels are collected, and after noise construction at the same iterative time step t, the target defect noise feature x that conforms to the Gaussian distribution is obtained. t ; Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t ; The target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain fine-grained texture detail information at the image level; the category text data T representing the target defect mask category is passed through the text feature encoder E T Converted into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , perform targeted sequence information conversion according to different texts to obtain serialized text vectors.

6. The method according to claim 1, characterized in that The method further includes: using the pre-training task data set to pre-train a Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

7. The method according to claim 5, characterized in that The Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition coding layer and three up-sampling sequence feature decoding modules; The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature, both of which have the same spatial dimension. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t ; The transition coding layer further distinguishes the regional information of the visual feature prediction sequence by adjusting the feature dimension, including defect part features and background general features; The three up-sampled sequence feature decoding modules are used to perform cubic interpolation up-sampling on the visual feature prediction sequence and perform Gaussian noise constraint; the self-attention layer output features in the three up-sampled sequence feature decoding modules are combined into a reference defect image visual feature sequence Or target defect image visual feature sequence The output sequences of the three up-sampling sequence feature decoding modules are used as detail information constraint sequences n t ; The serialized text vector and time step embedding vector are fed into the cross-fusion layer of each scale module of the network for fusion and embedding, respectively realizing text guidance and time step specification.

8. The method according to claim 7, characterized in that In the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment, specifically including: visual feature prediction sequence m t After being mapped to the visual feature constraint sequence f by a multi-layer perceptron t The same dimensional space is then used to calculate the negative cosine similarity, and the constraint control is performed using the pre-set adjustable constraint parameter λ.

9. The method according to claim 7, characterized in that In the detail information constraint module, the linear transformation layer is used to constrain the acquired detail information sequence n t Perform dimensionality reduction, and then use matrix dimension transformation to convert one-dimensional data into two-dimensional data; obtain the defect mask for detail information constraint by interpolation upsampling and sequence splicing, normalize the channels by taking the mean, and compare it with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

10. The method according to claim 7, characterized in that In the association space alignment module, the reference defect image visual feature sequence and the target defect image visual feature sequence Perform correlation alignment in the sequence correlation space, specifically including: The reference defect image visual feature sequence After the linear layer is mapped to the visual feature sequence of the target defect image The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication, and the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication is performed to obtain the target defect feature portion X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimension is transformed to obtain X' map ; After passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the predicted defect mask M is obtained P As the final prediction output, and in the training stage by comparing it with the target defect mask M T Calculate the binary cross entropy as the loss function for constraint.

Images