Strip steel defect segmentation method based on Markov reconstruction and correlation space alignment

Abstract

The invention provides a strip steel defect segmentation method based on Markov reconstruction and correlation space alignment, and relates to the technical field of strip steel surface defect detection. The method comprises the following steps: acquiring original images of the surface of the strip steel, wherein the original images comprise defect-free images and defect images of different types; marking the acquired image, and dividing a data set; respectively constructing an input data group in the pre-training task and the downstream task; performing feature extraction on the input data set to obtain an embedded feature vector of the input strip steel surface defect semantic segmentation base model; and constructing a strip steel surface defect semantic segmentation base model for strip steel surface defect segmentation. The strip steel surface defect semantic segmentation base model comprises two weight-shared Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module and a correlation space alignment module. According to the method, high-precision semantic segmentation can be realized for the defect sample which is never trained in a brand new scene.

Description

Technical Field

[0001] The invention relates to the technical field of strip steel surface defect detection, and in particular to a strip steel defect segmentation method based on Markov reconstruction and correlation space alignment. Background Art

[0002] Strip steel is an indispensable and important material in industry and is widely used in many fields such as automobiles, ships, and aerospace. However, during the hot rolling process, due to the limitations of the production process and equipment, defects such as cracks, scratches, and patches often appear on the surface of the strip steel. These surface defects not only affect the wear resistance, corrosion resistance, and fatigue strength of the steel, but also have a serious impact on the overall quality and service life of the product. Therefore, in order to ensure safe use and avoid potential risks, strict surface inspection must be carried out during the strip steel production process to detect and deal with these problems in a timely manner.

[0003] The manual visual defect detection method for high-speed moving strip steel has problems such as high work intensity, low efficiency, high missed detection rate and high false detection rate. Traditional machine vision-based detection technology has also been replaced by detection technology based on artificial intelligence deep learning due to its limitations such as poor generalization and difficulty in feature construction: the surface image of the strip steel is collected by a high-resolution camera, and the surface of the strip steel is segmented at a high-precision pixel level based on the semantic segmentation method to locate the defect area and realize fine-grained defect recognition.

[0004] However, current semantic segmentation methods have the following problems:

[0005] (1) The existing pre-training mechanism of the strip surface image base model is based on fine-grained texture learning or representation space alignment. These two training paradigms can only constrain parameter learning from one perspective and cannot fully explore the background regularity and defect knowledge of the strip image during the pre-training process. Considering that the strip surface images of the same batch have similar backgrounds but variable defects, as the base model for the downstream high-precision fine-grained semantic segmentation task, its limitations restrict the further improvement of the subtask accuracy.

[0006] (2) The current small sample downstream training paradigm is mainly based on the construction of adapters to adjust parameter weights, which cannot fully explore the characteristics of new scene samples for feature alignment. Compared with the natural image field, industrial scenes have challenges of new categories and complex samples. The current methods have limited improvement in the fine-grained segmentation performance of new scene and new defect samples.

[0007] (3) The existing target sample information extraction strategy cannot capture the key features of the defective part of the image to be identified, and does not fully utilize the reference new scene samples to extract associations between the images to be identified. The existing feature matching method feeds the segmentation results of the new scene samples into the network as mask features for learning, which cannot guide the extraction of visual features with the key information of the category, resulting in poor segmentation task results.

[0008] (4) The existing downstream fine-tuning method requires a new defect category to be given to train an additional decoder head, and the model is used to match the image to be identified with the learned model parameters for segmentation. However, due to the differences in the morphology of new defect categories and the lack of a large amount of fine-tuning data in actual industrial conditions, the generalization ability and segmentation quality of the model are affected.

[0009] (5) Finally, considering the strict product quality requirements of strip steel production, it is difficult to collect sufficient strip steel surface defect data in a completely new environment. The sparse training data makes it easy for the semantic segmentation model to overfit when fine-tuning the training in the new scenario. In addition, for defect forms that have never appeared before, the model lacks the features and knowledge learning of large-scale training data, making it difficult to effectively extract image information for high-precision segmentation.

[0010] Therefore, there is an urgent need for a strip surface defect segmentation method that can meet the needs of high-precision semantic segmentation without training in new scenarios. Summary of the invention

[0011] In view of the above problems, the purpose of the present invention is to provide a strip defect segmentation method based on Markov reconstruction and correlation space alignment, which can achieve high-precision semantic segmentation for defect samples that have never been trained in a new scenario.

[0012] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0013] On the one hand, a strip defect segmentation method based on Markov reconstruction and correlation space alignment is provided, the method comprising the following steps:

[0014] S1. Collecting original images of the strip surface, including defect-free images and defect images of different categories;

[0015] S2, labeling the defect targets in the defect image, performing one-hot encoding on all defect images according to different categories, and treating each defect image and the corresponding one-hot encoding label as a defect data pair;

[0016] S3, taking all defect-free images as pre-training task datasets, and dividing all defect data pairs into downstream task training sets and downstream task test sets in proportion as the complete downstream task dataset;

[0017] S4. In the pre-training task and downstream task, construct input data groups respectively;

[0018] In the pre-training task, each input data set includes a defect-free image and the corresponding text description; in the downstream task, each input data set includes the target defect image and target defect mask, the reference defect image and reference defect mask, the iteration time step and the category text data;

[0019] S5, extracting features from the input data group to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model;

[0020] S6. Constructing a strip steel surface defect semantic segmentation base model for strip steel surface defect segmentation;

[0021] The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module;

[0022] The obtained embedded feature vector is sent to two weight-shared Markov noise transfer matrix reconstruction networks, and the visual feature constraint module and the detail information constraint module are used to perform targeted constraints on the network training process from two aspects of visual features and detail information; the association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, matches the association areas on the target defect image and the reference defect image, and obtains the final prediction result.

[0023] Optionally, in step S2, all defect images are one-hot encoded according to different categories, including 9 defect categories: inclusions, scratches, holes, red iron sheets, iron sheet gray, plate oxide scale, roller marks, water stains and oil stains; the pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background and marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

[0024] Optionally, in step S3, all defect-free images are used as pre-training task data sets, and the text description of each defect-free image is ; at the same time, all defect data pairs are divided into a downstream task training set and a downstream task test set in a ratio of 8:2 as a complete downstream task data set.

[0025] Optionally, in the step S4, in the pre-training task, each input data group includes a defect-free image and a corresponding text description, which are used to pre-train the strip surface defect semantic segmentation base model;

[0026] In the downstream task, each input data set includes the target defect image IT , target defect mask M T , reference defect image I S , refer to defect mask M S , iterate time step t and category text data T;

[0027] Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect data set R∈N, where N is the number of all defect data pairs of this category, and the current defect image is removed. A defect data pair is randomly selected from the subset R'∈N-1 of the optional defect data set to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

[0028] Optionally, in step S5, feature extraction is performed on the input data group, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector;

[0029] The step S5 specifically includes:

[0030] For the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix dot multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is sent to the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iteration time step t t ;

[0031] The iterative time step t is passed through the linear embedding layer to increase the dimension and obtain the time step embedding vector y t , used for time step indication in feature space; target defect image I T After comparing with the reference defect image IS After the same data preprocessing operation, they are sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features are collected at different levels, and noise is constructed after iterative time step t to obtain the target defect noise feature x that conforms to the Gaussian distribution. t ; Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t ;

[0032] The target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain fine-grained texture detail information at the image level; the category text data T representing the target defect mask category is passed through the text feature encoder E T Transformed into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , according to different texts, targeted sequence information conversion is performed to obtain serialized text vectors.

[0033] Optionally, the method further comprises: using the pre-training task data set to pre-train a Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

[0034] Optionally, the Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition coding layer and three up-sampling sequence feature decoding modules;

[0035] The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature, both of which have the same spatial dimension. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t ;

[0036] The transition coding layer further distinguishes the regional information of the visual feature prediction sequence, including defect part features and background general features, by adjusting the feature dimension;

[0037] The three up-sampling sequence feature decoding modules are used to perform three-time interpolation up-sampling on the visual feature prediction sequence, and Gaussian noise constraints are performed; the self-attention layer output features in the three up-sampling sequence feature decoding modules are combined into a reference defect image visual feature sequence Or the visual feature sequence of the target defect image The output sequences of the three up-sampling sequence feature decoding modules are taken as detail information constraint sequences n t ;

[0038] The serialized text vector and time step embedding vector are sent to the cross-fusion layer of each different scale module of the network for fusion and embedding to achieve text guidance and time step specification respectively.

[0039] Optionally, in the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment, specifically including: visual feature prediction sequence m t After being mapped by a multi-layer perceptron to the visual feature constraint sequence f t The same dimensional space is then used to calculate the negative cosine similarity, and the constraint control is performed using a preset adjustable constraint parameter λ.

[0040] Optionally, in the detail information constraint module, a linear transformation layer is used to constrain the acquired detail information sequence n t Dimensionality reduction is performed, and then the one-dimensional data is converted into two-dimensional data by matrix dimension transformation; the defect mask for detail information constraint is obtained by interpolation upsampling and sequence splicing, and the channel is normalized by taking the mean and compared with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

[0041] Optionally, in the association space alignment module, the reference defect image visual feature sequence and the visual feature sequence of the target defect image Align correlation in sequence correlation space, including:

[0042] The visual feature sequence of the reference defect image Mapped to the visual feature sequence of the target defect image through the linear layer The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication, and the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication, to obtain the target defect feature part X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimensions are transformed to obtain X' map ; After passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the predicted defect mask M is obtained P As the final prediction output, and in the training phase, it is compared with the target defect mask M T Binary cross entropy is calculated as the loss function for constraint.

[0043] In another aspect, an electronic device is provided, the electronic device comprising:

[0044] processor;

[0045] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the strip defect segmentation method based on Markov reconstruction and association space alignment are implemented.

[0046] On the other hand, a computer-readable storage medium is provided, in which a program code is stored. The program code can be called by a processor to execute the steps of the strip defect segmentation method based on Markov reconstruction and correlation space alignment as described above.

[0047] The beneficial effects brought about by the technical solution provided by the present invention include at least:

[0048] (1) The present invention provides a powerful segmentation model pre-training constraint based on the Markov reconstruction training method, which can decode and reconstruct the pixels of the entire image and analyze the detail information. Through feature prediction and alignment constraints, the model can focus on capturing high-frequency information and key information in the image, further enhancing the base model's ability to understand the visual features of the strip surface image.

[0049] (2) The associative space alignment module of the present invention based on matrix calculation can perform associative calculation on the visual features of the specified layer and use a strict training task constraint model to mine the subtle associations between features, thereby realizing the prediction of similar defect areas between new defect samples and images to be segmented, thereby improving the model's accurate segmentation capability.

[0050] (3) Based on the matching mode of the reference defect image, the present invention enables the model to achieve accurate segmentation of the target defect to be predicted in a new scenario without training, and automatically match the associated areas on the image. Given a defect text category, similar feature pairing is performed by relying on only one reference sample and its mask, which enhances the practicality and generalization of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 is a schematic diagram of a strip defect segmentation method based on Markov reconstruction and correlation space alignment provided by an embodiment of the present invention;

[0053] Figure 2It is a schematic diagram of a feature extraction and input strip surface defect semantic segmentation base model provided by an embodiment of the present invention;

[0054] Figure 3 is a schematic diagram of a Markov noise transfer matrix reconstruction network provided by an embodiment of the present invention;

[0055] Figure 4 is a schematic diagram of a visual feature constraint module provided by an embodiment of the present invention;

[0056] Figure 5 is a schematic diagram of a detail information constraint module provided by an embodiment of the present invention;

[0057] Figure 6 is a schematic diagram of an associative space alignment module provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the word "exemplary" is used to present concepts in a specific way.

[0060] The embodiment of the present invention provides a strip defect segmentation method based on Markov reconstruction and correlation space alignment, such as Figure 1 As shown, the overall process of this method is as follows:

[0061] In the scenario of semantic segmentation of strip surface defects, a visible light camera is used to collect RGB strip surface images. After fine-grained and high-precision annotation, the data set is divided according to the proportion. The obtained strip surface defect data set is for each target defect image that needs to be segmented in the current scenario. Its input data group includes a reference defect image and corresponding mask for segmentation reference, a target defect image and corresponding mask, text data of the defect category of the current target, and Markov chain noise iteration time step. The above data is sent to the data feature extraction and embedding combined network, and the specified encoder is used to extract features from the image, text, and time step, and encode them into embedded feature vectors that can be fused and aligned. Subsequently, the obtained embedded feature vector is sent to the strip surface defect semantic segmentation base model.

[0062] The base model for semantic segmentation of surface defects of strip steel includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module. In the Markov noise transfer matrix reconstruction network, texture information is extracted through convolution operations, self-attention operations focus on important semantic parts of the image, the cross-fusion layer aligns text and visual information, and focuses on the defect area information of the multi-scale receptive field on the image to obtain the fine-grained texture features of the specified image. The network uses four loss functions to strictly constrain the network's ability to mine information and the effectiveness of vector features for high-precision segmentation. Gaussian noise prediction is used to indirectly implement the image reconstruction paradigm, and resampling reduces the difficulty of model fitting and improves the convergence speed.

[0063] In addition, the training task of the base model is constrained from the two aspects of image texture information and feature sequence space through the visual feature constraint module and the detail information constraint module, respectively, to improve the generalization of the network in downstream defect recognition. Finally, the association space alignment module is designed to align the visual feature sequences of the reference defect and the target defect in the sequence association space, obtain the defect area on the target defect image that is similar to the reference defect image, and use the constraints of the cross entropy loss function to force the model to learn the association between the visual feature sequences of the two images, thereby achieving high-precision semantic segmentation without training in new scenes and new defects downstream.

[0064] The method specifically comprises the following steps:

[0065] Step 1: Collect original images of the strip surface, including defect-free images and defect images of different categories.

[0066] In this step, the original images of the strip surface are captured by an RGB camera, and 12,000 defect-free images and 4,000 defective images of different categories are collected.

[0067] Step 2: Label the defect targets in the defect image, perform one-hot encoding on all defect images according to different categories, and take each defect image and the corresponding one-hot encoding label as a defect data pair.

[0068] In this step, all defect images are one-hot encoded according to different categories, including inclusions, scratches, holes, red iron sheets, iron sheet gray, plate oxide scale, roller marks, water stains and oil stains. The pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background and marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

[0069] Step 3: Take all defect-free images as pre-training task datasets, and divide all defect data pairs into downstream task training sets and downstream task test sets in proportion as the complete downstream task dataset.

[0070] In this step, all defect-free images are used as pre-training task datasets, and the text description of each defect-free image is . At the same time, all defect data pairs are divided into downstream task training sets and downstream task test sets in a ratio of 8:2 as a complete downstream task dataset. The model will learn the association of the defect targets to be identified on the downstream task training set, and verify the segmentation effect of the learned model under new defects and new scenarios on the downstream task test set.

[0071] Step 4: Construct input data groups in pre-training tasks and downstream tasks respectively.

[0072] In the pre-training task, each input data set includes a defect-free image and a corresponding text description, which is used to pre-train the strip surface defect semantic segmentation base model. By feeding large-scale defect-free strip surface images, the model can deeply mine image texture information in the Markov reconstruction task, learn to fit similar image distributions, and obtain the ability to extract visual features of downstream strip surface defect images.

[0073] In the downstream task, an input data set is constructed for each original defect data pair. Each input data set includes the target defect image I T , target defect mask M T , reference defect image I S , refer to defect mask M S , iterate time step t and category text data T;

[0074] Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect data set R∈N, where N is the number of all defect data pairs of this category, and the current defect image is removed. A defect data pair is randomly selected from the subset R'∈N-1 of the optional defect data set to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

[0075] Step 5: Perform feature extraction on the input data group to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model.

[0076] In this step, feature extraction is performed on the input data group, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector.

[0077] like Figure 2 As shown, for the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix dot multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is sent to the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iteration time step t t .

[0078] At the same time, the iterative time step t is upgraded through the linear embedding layer to obtain the time step embedding vector y t , used for time step indication in feature space. Target defect image I T After comparing with the reference defect image I S After the same data preprocessing operation, they are sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features are collected at different levels, and noise is constructed after iterative time step t to obtain the target defect noise feature x that conforms to the Gaussian distribution. t . Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t .

[0079] At the same time, the target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain the fine-grained texture detail information at the image level. In addition, the category text data T representing the target defect mask category (such as <an image of a strip steel surface with a scratch defect>) is passed through the text feature encoder E TTransformed into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , according to different texts, targeted sequence information conversion is performed to obtain a serialized text vector with a dimension of 1024.

[0080] Step 6. Construct a strip surface defect semantic segmentation base model for strip surface defect segmentation.

[0081] The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module and an associated space alignment module.

[0082] The obtained embedded feature vector is sent to two weight-shared Markov noise transfer matrix reconstruction networks, and the visual feature constraint module and the detail information constraint module are used to perform targeted constraints on the network training process from the two aspects of visual features and detail information; the association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, and matches the association area (i.e., the same defect part) on the target defect image and the reference defect image, and this area is the final prediction result. The target defect mask is used as the pixel-level constraint of the result, and is measured using the binary cross entropy function (BCE).

[0083] Step 7: Use the pre-training task data set to pre-train the Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

[0084] The present invention performs large-scale pre-training of the base model based on the Markov reconstruction paradigm, fully improves the base model's ability to extract visual features for the surface image of the strip, and sends the text information of the defect-free image into the guidance, enhancing the ability of the model text to align with the image, so that the text sequence can better realize the semantic guidance of different defect categories in the downstream. The present invention constructs a Markov chain to model the image distribution, and realizes image reconstruction based on the probabilistic graphical model by gradually adding and removing the noise transfer matrix on the original image. Set the total time step T to 1000, and consider the transfer from the original image x0 to the state x close to the Gaussian distribution t It can be obtained by sampling through formula (1). Therefore, a parameterized reverse Markov chain is designed to gradually transfer from the standard Gaussian distribution to the original image by gradually removing the noise, that is, reconstructing the original image. In order to obtain a reverse transfer matrix that is close to the distribution of the forward noise transfer matrix, the Markov noise transfer matrix reconstruction network D is trained for prediction. The noise transfer matrix ε predicted by the network is D (x t,t) and the noise transfer matrix ε added at the corresponding time step are constrained by the mean square error (MSE).

[0085]

[0086] Where x0 is the original image, x t is the image after adding Gaussian noise.

[0087]

[0088] α i is the noise scheduling parameter, in

[0089] Step 8. Figure 3 As shown, the Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition coding layer and three up-sampling sequence feature decoding modules.

[0090] The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature. The spatial dimensions of the two are the same, and two identical networks with shared weights can be used. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t . Visual feature prediction sequence m t There are three dimensions (three channels), which can be written as Where i=1,2,3, and the three dimensions are [32*32*256], [16*16*512] and [8*8*1024] respectively.

[0091] The transition coding layer further distinguishes the regional information of the visual feature prediction sequence by adjusting the feature dimension, including the defect part features and the background general features. The three upsampling sequence feature decoding modules with the same number of channels and dimension information as the downsampling sequence feature encoding module are used to perform three-time interpolation upsampling on the visual feature prediction sequence and perform Gaussian noise constraints. Predicting Gaussian noise Label with Gaussian noise Constraint is performed by the mean square error (MSE).

[0092] At the same time, the visual sequences of different nodes in the upsampling decoding process are taken out, constrained and further processed respectively to improve the accuracy and generalization of the network's prediction of segmentation masks under new defects. The output features of the self-attention layer in the three upsampling sequence feature decoding modules are combined into the visual feature sequence of the reference defect image. Or the visual feature sequence of the target defect image and There are three dimensions, which can be written as Where i = 1, 2, 3, the dimensions are [64*1280], [256*1280] and [1024*1280] respectively. The output sequences of the three upsampling sequence feature decoding modules are taken as the detail information constraint sequence n t , n t There are three dimensions, which can be written as Where i=1,2,3, the dimensions are [8*8*1024], [16*16*512] and [32*32*256] respectively.

[0093] The serialized text vector and time step embedding vector are sent to the cross-fusion layer of each different scale module of the network for fusion and embedding to achieve text guidance and time step specification respectively.

[0094] Step 9, such as Figure 4 As shown, in the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment. The visual feature prediction sequence and the visual feature constraint sequence have three dimensions, which can be recorded as and Where i=1,2,3.

[0095] Specifically, the visual feature prediction sequence m t After being mapped by a multi-layer perceptron to the visual feature constraint sequence f t The same dimensional space is then used to calculate the negative cosine similarity and to perform constraint control using the preset adjustable constraint parameter λ. The loss function L similarity The measurement is shown in formula (2).

[0096]

[0097] Among them, λ is an adjustable constraint parameter, and its default value is 0.5.

[0098] Step 10, such as Figure 5 As shown, in the detail information constraint module, the acquired detail information constraint sequence n t It consists of features of three different dimensions. First, a linear transformation layer is used to constrain the sequence n of detail information. tThe dimensionality is reduced to [64*1], [256*1] and [1024*1]. The one-dimensional data is then converted into two-dimensional data using matrix dimension transformation to restore the image spatial information. Subsequently, all image visual features are interpolated and upsampled to 64*64, and multiple sequences are concatenated to obtain a defect mask of dimension [64*64*3] for detail information constraint. Finally, the channel is normalized to [64*64*1] by taking the mean, and compared with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

[0099] Step 11, such as Figure 6 As shown, in the association space alignment module, the reference defect image visual feature sequence and the visual feature sequence of the target defect image Correlation alignment is performed in the sequence association space, and the two visual feature sequences are registered at the defective part through matrix calculation of the same dimension. Specifically, it includes:

[0100] The visual feature sequence of the reference defect image Mapped to the visual feature sequence of the target defect image through the linear layer The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication; the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication, to obtain the target defect feature part X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimensions are transformed to obtain X' map , X′ map = reshape(mean(concat[X map i ])), i = 1, 2, 3; after passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the texture information of the currently obtained aligned feature map can be further enhanced to obtain the predicted defect mask M P As the final prediction output, and in the training phase, it is compared with the target defect mask M T Binary cross entropy is calculated as the loss function for constraint.

[0101] In an embodiment of the present invention, a training paradigm based on Markov reconstruction is designed to address the problem that the existing segmentation model pre-training strategy is difficult to constrain the semantic segmentation of strip steel images at both the visual feature level and the detail information level. A visual feature constraint module and a detail information constraint module are designed based on the output features of different network modules to achieve learning of fine-grained pixel information and feature associations of the entire image, thereby improving the feature extraction capability of the base model.

[0102] In view of the problem that the existing target sample information extraction strategy cannot capture the key features of the part that needs to be segmented in the image to be identified, the present invention proposes a correlation space alignment module for the image visual feature sequence, which explores the correlation information between the reference new defect samples and the image to be segmented in the feature correlation space, and uses the text information of the defect category to realize the prior guidance of the reference defect image, thereby further improving the high-precision segmentation capability of new defects.

[0103] In order to solve the problem that the existing base model still needs a certain scale of training samples and resource requirements for fine-tuning in the face of new scenarios and new defects after pre-training with large-scale data and large-volume parameter structures, the present invention designs a downstream prediction system for image-specified defect recognition and segmentation based on defect samples. In a completely new scenario, the defect area is segmented by feature pairing, thereby achieving accurate segmentation of new samples without training.

[0104] In an exemplary embodiment, the present invention further provides an electronic device, the electronic device comprising:

[0105] processor;

[0106] A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are loaded and executed by the processor, the steps of the strip defect segmentation method based on Markov reconstruction and association space alignment are implemented.

[0107] In an exemplary embodiment, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the computer-readable storage medium, and the at least one instruction is loaded and executed by a processor to implement the steps of the strip defect segmentation method based on Markov reconstruction and association space alignment as described above. For example, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0108] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0109] References in the specification to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments", etc. indicate that the described embodiments may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. In addition, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the relevant art to implement such feature, structure, or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0110] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0111] In the present invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0112] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0113] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0114] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0115] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0116] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0117] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A strip defect segmentation method based on Markov reconstruction and correlation space alignment, characterized in that: The following steps are involved: S1. Collecting original images of the strip surface, including defect-free images and defect images of different categories; S2, labeling the defect targets in the defect image, performing one-hot encoding on all defect images according to different categories, and treating each defect image and the corresponding one-hot encoding label as a defect data pair; S3, taking all defect-free images as pre-training task datasets, and dividing all defect data pairs into downstream task training sets and downstream task test sets in proportion as the complete downstream task dataset; S4. In the pre-training task and downstream task, construct input data groups respectively; In the pre-training task, each input data set includes a defect-free image and the corresponding text description; in the downstream task, each input data set includes the target defect image and target defect mask, the reference defect image and reference defect mask, the iteration time step and the category text data; S5, extracting features from the input data group to obtain an embedded feature vector of the input strip surface defect semantic segmentation base model; S6. Constructing a strip steel surface defect semantic segmentation base model for strip steel surface defect segmentation; The strip surface defect semantic segmentation base model includes: two weight-sharing Markov noise transfer matrix reconstruction networks, a visual feature constraint module, a detail information constraint module, and an associated space alignment module; The obtained embedded feature vector is sent to two weight-shared Markov noise transfer matrix reconstruction networks, and the visual feature constraint module and the detail information constraint module are used to perform targeted constraints on the network training process from two aspects of visual features and detail information; the association space alignment module aligns the visual feature sequences corresponding to the vector dimensions and information in the two networks in the association space, matches the association areas on the target defect image and the reference defect image, and obtains the final prediction result.

2. The method according to claim 1, characterized in that In step S2, all defect images are one-hot encoded according to different categories, including 9 defect categories: inclusions, scratches, holes, red iron sheets, iron sheet gray, plate oxide scale, roller marks, water stains and oil stains; the pixel points corresponding to the defect target on each defect image are marked as <1> , the rest are set as background and marked as <0> ; Treat each defect image and the corresponding one-hot encoded label as a defect data pair.

3. The method according to claim 1, characterized in that In step S3, all defect-free images are used as pre-training task data sets, and the text description of each defect-free image is ; at the same time, all defect data pairs are divided into a downstream task training set and a downstream task test set in a ratio of 8:2 as a complete downstream task data set.

4. The method according to claim 1, characterized in that In the step S4, in the pre-training task, each input data group includes a defect-free image and a corresponding text description, which is used to pre-train the strip surface defect semantic segmentation base model; In the downstream task, each input data set includes the target defect image I T , target defect mask M T , reference defect image I S , refer to defect mask M S , iterate time step t and category text data T; Among them, the target defect image I T and target defect mask M T That is, the current defect image and its unique hot encoding label in the defect data pair; all defect data pairs of the defect category to which the current defect image belongs are regarded as the optional defect data set R∈N, where N is the number of all defect data pairs of this category, and the current defect image is removed. A defect data pair is randomly selected from the subset R'∈N-1 of the optional defect data set to form a reference defect data pair, and the defect image of the reference defect data pair is used as the reference defect image I S , whose corresponding one-hot encoded label is used as the reference defect mask M S ; For each defect category, provide its category text as the category text data T in the input data group; for each current input data group, provide the iterative time step t required for the Markov noise transfer matrix to reconstruct the network, t∈[0,1000).

5. The method according to claim 1, characterized in that In the step S5, feature extraction is performed on the input data group, and the obtained embedded feature vector includes: reference defect noise feature, target defect noise feature, visual feature constraint sequence, time step embedding vector, and serialized text vector; The step S5 specifically includes: For the reference defect image I S and the reference defect mask M S Randomly crop and standardize the reference defect image of RGB three channels, and then S With M S Perform matrix dot multiplication, retain the defect part on the reference defect image, and set the background part to <0> ; The processed reference defect image is sent to the variational image autoencoder E V , extract the original feature x o ; After iterative time step t, noise is constructed and the original feature x o Gaussian noise is added to each pixel value to obtain the reference defect noise feature x' that conforms to the Gaussian distribution corresponding to the iteration time step t t ; The iterative time step t is passed through the linear embedding layer to increase the dimension and obtain the time step embedding vector y t , used for time step indication in feature space; target defect image I T After comparing with the reference defect image I S After the same data preprocessing operation, they are sent to the variational image autoencoder E V and image sequence feature extractor E C Visual features are collected at different levels, and noise is constructed after iterative time step t to obtain the target defect noise feature x that conforms to the Gaussian distribution. t ; Through the image sequence feature extractor E C , after downsampling and convolution operations, a visual feature constraint sequence f with multi-scale semantic information of defective images is obtained t ; The target defect mask M marked with the target defect T After cropping and scaling at the resolution scale, it is used to constrain fine-grained texture detail information at the image level; the category text data T representing the target defect mask category is passed through the text feature encoder E T Transformed into a sequence with corresponding semantic information in feature space, and then sent to the learnable linear layer L T , according to different texts, targeted sequence information conversion is performed to obtain serialized text vectors.

6. The method according to claim 1, characterized in that The method also includes: using the pre-training task data set to pre-train the Markov noise transfer matrix reconstruction network in the strip surface defect semantic segmentation base model.

7. The method according to claim 5, characterized in that The Markov noise transfer matrix reconstruction network includes three down-sampling sequence feature encoding modules, a transition encoding layer and three up-sampling sequence feature decoding modules; The backbone input of the network is the reference defect plus noise feature or the target defect plus noise feature, both of which have the same spatial dimension. After interpolation and downsampling by the three downsampling sequence feature encoding modules, the output visual features of different scales constitute the visual feature prediction sequence m t ; The transition coding layer further distinguishes the regional information of the visual feature prediction sequence, including defect part features and background general features, by adjusting the feature dimension; The three up-sampling sequence feature decoding modules are used to perform three-time interpolation up-sampling on the visual feature prediction sequence, and Gaussian noise constraints are performed; the self-attention layer output features in the three up-sampling sequence feature decoding modules are combined into a reference defect image visual feature sequence Or the visual feature sequence of the target defect image The output sequences of the three up-sampling sequence feature decoding modules are taken as detail information constraint sequences n t ; The serialized text vector and time step embedding vector are sent to the cross-fusion layer of each different scale module of the network for fusion and embedding to achieve text guidance and time step specification respectively.

8. The method according to claim 7, characterized in that In the visual feature constraint module, the obtained visual feature prediction sequence m t The visual feature constraint sequence f t Alignment, specifically including: visual feature prediction sequence m t After being mapped by a multi-layer perceptron to the visual feature constraint sequence f t The same dimensional space is then used to calculate the negative cosine similarity, and the constraint control is performed using a preset adjustable constraint parameter λ.

9. The method according to claim 7, characterized in that: In the detail information constraint module, a linear transformation layer is used to obtain the detail information constraint sequence n. t Dimensionality reduction is performed, and then the one-dimensional data is converted into two-dimensional data by matrix dimension transformation; the defect mask for detail information constraint is obtained by interpolation upsampling and sequence splicing, and the channel is normalized by taking the mean and compared with the target defect mask M T Calculate the binary cross entropy to realize the detail information constraint of the texture level of the defect image.

10. The method according to claim 7, characterized in that In the association space alignment module, the reference defect image visual feature sequence and the visual feature sequence of the target defect image Align correlation in sequence correlation space, including: The visual feature sequence of the reference defect image Mapped to the visual feature sequence of the target defect image through the linear layer The same dimensional space, get the dimension matrix V S , and the dimension matrix V of the target defect image visual feature sequence Q The correlation is calculated by matrix multiplication, and the correlation degree of each sequence value is obtained through the softmax function layer, and then compared with V S Matrix multiplication, to obtain the target defect feature part X related to the reference defect feature map ; for X map The features of the three dimensions are averaged and the matrix dimensions are transformed to obtain X' map ; After passing through the 1*1 convolution layer and the interpolation upsampling convolution layer, the predicted defect mask M is obtained P As the final prediction output, and in the training phase, it is compared with the target defect mask M T Binary cross entropy is calculated as the loss function for constraint.

Images