Steel image defect detection method in industrial production based on self-supervised contrastive representation learning technology

Through self-supervised comparison characterization learning technology, the use of labelless data for steel image defect detection, combined with EMD distance measurement, the problem of relying on a large amount of labeled data in the existing technology is solved, and the efficient defect detection effect in industrial production is achieved.

CN114240886BActive Publication Date: 2024-11-29HEFEI UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202111551815.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-17
Publication Date
2024-11-29
Estimated Expiration
2041-12-17

AI Technical Summary

Technical Problem

The prior art relies on a large amount of labeled data in the detection of steel picture defects, which is difficult to apply in industrial production, especially when acquiring labeled data is high cost and inefficient.

Method used

Using self-supervised contrast characterization learning technology, the upstream steel surface defect characterization learning model and downstream detector are constructed, and self-supervised training is performed using labelless data. Combined with EMD distance measurement, the structure and position information of the defect are retained to reduce the impact of contrast learning noise. .

Benefits of technology

Without the need for a large amount of labeled data, the efficient performance of steel image defect detection is achieved, the threshold for deep learning technology in the industrial field is reduced, and the defect detection effect is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114240886B_ABST
    Figure CN114240886B_ABST
Patent Text Reader

Abstract

The present invention relates to a method for detecting defects in steel images in industrial production based on self-supervised contrastive representation learning technology. Compared with the prior art, it solves the defect that steel image defect detection needs to rely on a large amount of labeled data. The present invention includes the following steps: acquisition and preprocessing of industrial production steel images; construction of an upstream steel surface defect representation learning model; training of an upstream steel surface defect representation learning model; construction of a downstream steel surface defect detector; training of a downstream steel surface defect detector; acquisition of industrial production steel images to be detected; and acquisition of industrial production steel image defect detection results. The present invention does not require a large amount of labeled data, and obtains a good representation of defect data by performing self-supervised representation learning on a large amount of unlabeled data sets, thereby realizing steel image defect detection in industrial production using a small amount of labeled steel defect images.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of industrial image detection, and in particular to a method for detecting steel image defects in industrial production based on a self-supervised contrast representation learning technology. Background Art

[0002] Defect detection is an important part of industrial production. Steel surface defect detection refers to the detection of scratches, cracks, foreign matter, corrosion, holes and other problems on the surface of steel. Traditional detection methods that rely on manpower have disadvantages such as low efficiency and the detection results are easily affected by human subjective factors. With the advancement of science and technology, some non-destructive testing methods have been proposed and applied to various defect detection fields. In the field of steel defect detection, magnetic particle testing, penetrant testing, eddy current testing, ultrasonic testing, X-ray and other technologies are mainly used. With the vigorous development of artificial intelligence technology, many new defect detection methods have been proposed and applied to industrial environments.

[0003] Some steel defect detection methods based on traditional visual technology have played a huge role for a long time. These methods are mainly divided into three categories: methods based on texture features, methods based on color features, and methods based on shape features. The rapid development of deep learning technology has also made it widely used in the field of steel defect detection. Many steel defect sample classification and detection technologies have been proposed, including supervised methods, unsupervised methods, and weakly supervised methods. This type of deep learning-based method is trained on a large number of steel surface defect samples to autonomously learn various types of steel defect characteristics. It has achieved good results in various industrial environments, has strong robustness, and can ensure the speed of detection to meet the needs of industrial production.

[0004] Although deep learning-based methods have the characteristics of high accuracy and strong robustness, the training of their models requires a large amount of data, especially labeled data. Steel surface defects have the characteristics of single color, small defect targets, and multiple defects appearing in parallel. It is difficult for non-professionals to distinguish and summarize various defect samples, resulting in a very high cost for obtaining labeled data in this field.

[0005] Since the defect detection model based on deep learning contains tens of millions of parameters, a large amount of labeled data is required to fit the parameters during model training. The lack of labeled data will greatly affect the steel defect detection effect based on deep learning methods. This also makes it impossible to apply traditional deep learning defect detection technology to steel image detection in industrial production, because in actual industrial production, it is impossible to obtain a large number of steel defect images (labeled samples), and there are only a very small number of steel defect images. Unlabeled samples are difficult to achieve actual use effects in traditional deep learning detection technology.

[0006] Therefore, how to propose a method to detect steel image defects without relying on a large amount of labeled data has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] The purpose of the present invention is to solve the defect that steel image defect detection in the prior art requires reliance on a large amount of labeled data, and to provide a steel image defect detection method in industrial production based on self-supervised contrast representation learning technology to solve the above problem.

[0008] In order to achieve the above object, the technical solution of the present invention is as follows:

[0009] A method for detecting defects in steel images in industrial production based on self-supervised contrastive representation learning technology comprises the following steps:

[0010] Acquisition and preprocessing of industrial steel images: Acquisition of labeled and unlabeled industrial steel images and preprocessing;

[0011] Construction of upstream steel surface defect characterization learning model: Construction of upstream steel surface defect characterization learning model;

[0012] Training of upstream steel surface defect characterization learning model: Use pre-processed unlabeled industrial production steel images to train the upstream steel surface defect characterization learning model;

[0013] Construction of downstream steel surface defect detector: Based on the backbone feature extraction network obtained by training with a large number of unlabeled images, the downstream steel surface defect detector is constructed;

[0014] Training of downstream steel surface defect detectors: Use pre-processed labeled industrial production steel images to train downstream steel surface defect detectors;

[0015] Acquisition of images of industrial steel to be tested: Acquisition of images of industrial steel to be tested and preprocessing;

[0016] Obtaining defect detection results for industrial steel images: Input the preprocessed industrial steel images to be detected into the trained downstream steel surface defect detector to obtain defect detection results for the industrial steel images.

[0017] The acquisition and preprocessing of the industrial production steel pictures include the following steps:

[0018] Get pictures of industrially produced steel;

[0019] Data augmentation for industrial steel images:

[0020] First, all industrial steel pictures were resized to 224x224, and then random cropping and random horizontal flipping were applied; then brightness color enhancement with an intensity of 0.4, contrast enhancement with an intensity of 0.4, hue enhancement with an intensity of 0.1, application probability with an intensity of 0.8, and random grayscale transformation with an application probability of 0.2 were applied;

[0021] Gaussian blur processing is performed on the industrial production steel images after data enhancement.

[0022] The construction of the upstream steel surface defect characterization learning model includes the following steps:

[0023] Set the backbone feature extraction network ResNet-50 in SimSiam:

[0024] The global pooling layer in the backbone feature extraction network ResNet-50 in the original SimSiam is removed, and the obtained feature map is retained and continuously passed to the subsequent nonlinear mapping module in the contrastive learning framework; the encoder f without the pooling layer is used to extract features from the original unlabeled steel surface defect data, and the output is a feature map, denoted as Y;

[0025] A 1x1 convolution structure is used to replace the original MLP, the convolution structure is used to map the feature map, and the EMD distance with a matching mechanism is used to measure the similarity of various steel defects.

[0026] The training of the upstream steel surface defect characterization learning model includes the following steps:

[0027] The preprocessed unlabeled industrial steel image x is randomly augmented twice to obtain two augmented views x1 and x2:

[0028] x1=Augmentation random (x),x2=Augmentation random (x);

[0029] The two augmented views x1 and x2 are respectively input into the encoder f for feature extraction to obtain two sets of feature maps Y1 and Y2; the encoder f is a ResNet-50 network that discards the fully connected layer and the global pooling layer, and the encoder f shares weights between the two augmented views;

[0030] The two sets of feature maps Y1 and Y2 are further mapped using a 1x1 convolution with a sliding step size of 1, and the size of the feature maps is kept unchanged;

[0031] Add one more level of MLP mapping h on one side;

[0032] Considering the computational complexity, a pooling layer is added before this to pool the feature map into a one-dimensional vector for the final similarity calculation. The two vectors are represented as:

[0033] and

[0034] Among them, f is the backbone feature extraction network, g is the first-level mapping module, which is composed of 1x1 convolution, and h is the second-level mapping module, which is composed of MLP; P1 is the feature vector obtained after x1 passes through the feature extraction network f and the subsequent two mapping modules g and h, and Z2 is the feature vector obtained after x2 passes through the feature extraction network f and the subsequent mapping module g;

[0035] Use EMD distance to calculate the similarity of two vectors:

[0036] The symmetric loss is adopted, that is, the results obtained by exchanging x1 and x2 through the network are expressed as Z1 and P2;

[0037] The EMD distance formula and loss function L are defined as follows:

[0038] D EMD (P,Z2)=||P-Z2|| EMD ,

[0039]

[0040] Among them, Stopgrad() stops the gradient backpropagation operation.

[0041] The construction of the downstream steel surface defect detector includes the following steps:

[0042] For the upstream steel surface defect characterization learning model, the mapping comparison module of the characterization model SimSE is abandoned to obtain the backbone feature extraction network ResNet-50;

[0043] Design a weight migration scheme: load the weights of the backbone feature extraction network part of SimSE obtained by contrastive representation learning on a large number of unlabeled steel surface defect datasets into the backbone feature extraction network ResNet-50 of the downstream steel surface defect detector Faster R-CNN by changing the weight name;

[0044] Faster R-CNN, which has been loaded with steel surface defect characterization weights, is used as the downstream steel surface defect detector.

[0045] The training of the downstream steel surface defect detector is to extract features from the input image through the backbone network to obtain the feature map of the image; then the feature map is input into the RPN proposal network to generate region proposals to obtain the proposal box; the ROI pooling layer generates the proposal feature map in the feature map output by the feature extraction network according to the region proposal of the RPN network; and then the generated proposal feature map is passed into the subsequent target category determination network and the bounding box regression network to obtain the final target category and the precise position of the detection box; the specific steps are as follows:

[0046] The input steel surface defect image is resized to 600x800 and passed to the backbone feature extraction network ResNet-50 for feature extraction to obtain a set of 38x50 shared feature layers;

[0047] The shared feature layer is passed into the RPN suggestion frame network to generate the suggestion frame for steel surface defect detection; the shared feature layer is convolved 3x3, and the result is convolved twice with 1x1. One of the 1x1 convolution channels has 18, and the result is used to determine whether the prior frame actually contains an object. The other 1x1 convolution channel has 36 channels, which is used to determine the four parameters of the suggestion frame, namely the x-axis and y-axis of the center point, and the height and width of the frame; after these two sets of convolutions, the prior frame is adjusted to the suggestion frame;

[0048] The proposed box obtained by the RPN network is combined with the original shared feature layer, and passed to the RoIAlign layer to intercept the shared feature layer. The RoIAlign layer performs regional pooling on all local feature layers to obtain local feature layers of the same size.

[0049] All the local feature layers obtained are passed to the subsequent classification and regression network for classification prediction and regression prediction respectively; the result of the regression prediction network will make fine adjustments to the previously obtained suggestion box to obtain an accurate final detection box; the classification prediction network uses the softmax function for mapping to obtain multi-classification probability values, judge the specific type of steel surface defects in the detection box, use the loss function to compare the obtained results with the true value, use the gradient descent method to adjust the network parameters, gradually reduce the value of the loss function, and finally obtain the trained downstream steel surface defect detector.

[0050] The design weight migration scheme includes the following steps:

[0051] Use Amodel_dict to load the weights of the upstream steel surface defect characterization learning model, and use Bmodel_dict to load the weights of the downstream steel surface defect detector;

[0052] Traverse the key-value pairs of the downstream steel surface defect detector weights, find the key name of the part to be loaded, replace the key name of the part of the upstream steel surface defect representation learning model with the key name of the part of the downstream steel surface defect detector, and use key_new to receive the changed key value;

[0053] At this time, Amodel_dict[key_new] is the part of the upstream steel surface defect characterization learning model that matches the downstream steel surface defect detector, that is, the same backbone feature extraction network ResNet-50 part;

[0054] The weights of the corresponding parts in the downstream steel surface defect detector are overwritten by the weights of the corresponding parts in the upstream steel surface defect characterization learning model.

[0055] The method of calculating the similarity of two vectors using the EMD distance comprises the following steps:

[0056] The two vectors obtained after the two augmented views pass through the feature extraction network and the mapping module are represented as P1={x i |i=1,2,3,...m},Z2={y j |j=1,2,3,...n}, the distance between the individual components of the vector is defined as: d(x i ,y j )=|x i -y i |, where x, y are the components of vectors P1 and Z2 respectively;

[0057] Define a coupling matrix S, which represents the minimum matching solution between the components of vectors P1 and Z2, and then define a distance matrix C, which is used to record the distance from each component of vector P1 to each component of vector Z2. That is, each element C of the matrix ij They represent the distance from the i-th vector of P1 to the j-th vector of Z2 respectively;

[0058] The EMD distance between two vectors is calculated using the Frobenius inner product between the solution matrix S and the distance matrix C. The calculation formula is as follows:

[0059]

[0060] Find the optimal matching solution to minimize the total distance. The optimization problem is defined as:

[0061]

[0062] stS1=P1

[0063] S T 1 = Z2;

[0064] The fast iterative method Sinkhorn-Knopp is used to solve the constrained optimization problem. The entropy of a matrix is ​​defined as The regularization coefficient ε is introduced to obtain a smoother coupling matrix:

[0065]

[0066] stS1=P1

[0067] P T 1 = Z2;

[0068] By increasing the regularization coefficient ε, the resulting coupling matrix is ​​smoother;

[0069] When ε approaches zero, the coupling matrix becomes more sparse, and the final solution is closer to the original optimal matching distance problem;

[0070] After introducing this entropy regularization, the problem is transformed into a convex optimization problem, which is then solved using the Sinkhorn-Knopp iteration. The optimal matching solution is further expressed as:

[0071] S = diag(u)Kdiag(v),

[0072] During the iteration, the sequences u and v are updated alternately:

[0073]

[0074] Among them, K is the kernel matrix, and its elements are calculated by the distance matrix C, expressed as:

[0075] The iterative algorithm is used to find the minimum match between the components of the two sets of vectors to obtain the optimal matching matrix S. Then, the minimum cost of transforming vector P1 into vector Z2 is obtained by calculating the Frobenius inner product of the coupling matrix S and the cost matrix C. The minimum cost is the EMD distance of the two vectors, which is expressed as:

[0076] D EMD (P1,Z2)=<C,S> ,

[0077] D EMD (P1,Z2)=||P1-Z2|| EMD ;

[0078] Using symmetric loss, that is, exchanging x1 and x2, the results obtained through the network are expressed as Z1 and P2, and the distance between the two vectors is calculated using EMD:

[0079] D EMD (P2,Z1)=||P2-Z1||EMD ,

[0080] The overall loss function is defined as follows:

[0081]

[0082] Among them, Stopgrad() stops the gradient backpropagation operation.

[0083] Beneficial Effects

[0084] The method for detecting steel image defects in industrial production based on self-supervised contrastive representation learning technology of the present invention does not require a large amount of labeled data (labeled data) compared with the prior art, and obtains a good representation of the defect data by performing self-supervised representation learning on a large amount of unlabeled data sets; at the same time, the original self-supervised model is improved to overcome the defect that the original model is not suitable for steel surface defect detection tasks, thereby realizing steel image defect detection in industrial production using a small amount of steel defect images.

[0085] (1) The technology proposed in this invention lowers the threshold for the application of defect detection based on deep learning technology in the industrial field, and can achieve excellent defect detection results in industrial environments where it is difficult to obtain labeled data;

[0086] (2) Different from the prior art which only extracts global semantic features in the defect characterization stage, the present invention pays special attention to and retains key features such as structure and position that can reflect the context information of various defects in the design of the upstream SimSE characterization learning model, so as to obtain better raw data representation, which is more conducive to the development of downstream defect detection tasks;

[0087] (3) Considering that images in the steel surface defect detection dataset generally contain multiple types of defect targets, if they are treated as the same object, noise will inevitably be introduced into the contrastive learning framework. We propose to use the EMD distance with a matching mechanism to measure similarity, eliminating the negative impact of the presence of multiple types of targets in the image on contrastive representation learning;

[0088] (4) A large number of experimental results show that after using unlabeled data to obtain a good representation of the defect object, the present invention only needs a small amount of labeled data to obtain a high-performance steel surface defect detector in the downstream detection task, and the detection effect is significantly better than the existing technical methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 is a method sequence diagram of the present invention;

[0090] Figure 2 It is the structure diagram of the upstream characterization model SimSE of the present invention;

[0091] Figure 3 is a structural diagram of a downstream defect detector of the present invention;

[0092] Figure 4 It is a schematic diagram of the overall technical framework of the present invention;

[0093] Figure 5 and Figure 6 All of them are schematic diagrams of defect detection effects of the method described in the present invention. DETAILED DESCRIPTION

[0094] In order to have a further understanding and recognition of the structural features and the effects achieved by the present invention, a preferred embodiment and accompanying drawings are used for detailed description as follows:

[0095] like Figure 1 As shown, the method for detecting steel image defects in industrial production based on self-supervised contrast representation learning technology described in the present invention comprises the following steps:

[0096] The first step is to obtain and preprocess industrial steel pictures: obtain labeled and unlabeled industrial steel pictures and preprocess them.

[0097] (1) Obtain pictures of industrially produced steel;

[0098] (2) Data enhancement processing of industrial steel production images:

[0099] First, all industrial steel pictures were resized to 224x224, and then random cropping and random horizontal flipping were applied; then brightness color enhancement with an intensity of 0.4, contrast enhancement with an intensity of 0.4, hue enhancement with an intensity of 0.1, application probability with an intensity of 0.8, and random grayscale transformation with an application probability of 0.2 were applied;

[0100] (3) Gaussian blur processing is performed on the industrial production steel images after data enhancement.

[0101] The second step is to construct an upstream steel surface defect characterization learning model: construct an upstream steel surface defect characterization learning model.

[0102] Here, a self-supervised contrastive representation learning (SimSE) defect detection technology framework that does not rely on a large amount of labeled data is proposed to realize the design of steel defect detector; based on contrastive learning, it is a self-supervised representation learning method, and its main purpose is to use artificially defined proxy tasks in upstream tasks to learn a good representation of the original data, so as to facilitate the subsequent downstream tasks.

[0103] The basic framework of self-supervised contrastive learning adopted is SimSiam. The purpose of self-supervised representation learning is to use a large amount of unlabeled data to generate better feature representations. It trains and improves the network's representation learning ability by maximizing the consistency between different transformed views of the same image and minimizing the consistency between different images.

[0104] The specific method of SimSiam is to use two random augmented views x1 and x2 of the input image x as the input of the model. The model extracts features from the input image through the encoder f. The encoder f is a ResNet-50 network without the fully connected layer, and f shares weights between the two augmented images. The two augmented views are passed through the encoder f and subsequent pooling operations (pooling) to obtain their respective vector representations y1 and y2. After that, the projection head g and the prediction head h are used to map the vector y1 obtained by passing the augmented view x1 through the encoder f and pooling twice to obtain vector representations z1 and p respectively. The projection head g is used alone to map the vector y2 obtained by passing the augmented view x2 through the encoder f and pooling to obtain z2. After that, the feature vectors p and z2 of the two augmented views are calculated for contrast loss in the feature space.

[0105] Although SimSiam can achieve good results in defect classification tasks after sufficient representation training on unlabeled datasets, this method is not suitable for steel surface defect detection tasks. First, because the steel defect detection task is extremely dependent on the structure and position information of various defects in the steel defect image, the SimSiam framework globally pools the feature map into a one-dimensional vector before projecting the MLP. This operation will lose the position and structure information of the steel defect data. Secondly, due to the mechanism problem of the contrastive learning framework, that is, it includes random cropping operations when performing random augmentation of images. In the steel surface defect detection dataset, a single defect image often contains multiple or multiple types of defect targets. After random cropping, different positions in the two augmented views will contain multiple different defect targets. However, the potential prior of contrastive learning is that different augmentations of the same image should correspond to the same object. The goal of contrastive learning is also to maximize the similarity of the vectors corresponding to the two augmented views. If the randomly cropped images contain different targets, the one-dimensional feature vectors obtained after multi-level mapping of the network will inevitably lose the contextual information. If these vectors are directly measured by the cosine similarity in the SimSiam model, it will violate the potential prior and is obviously very unreasonable.

[0106] In order to make the contrastive learning model SimSiam suitable for the migration of steel surface defect detection tasks, we adjusted the structure of SimSiam. Since the original SimSiam uses ResNet-50 to remove the fully connected layer and obtain a one-dimensional vector through global pooling for subsequent nonlinear mapping, and the vector has lost the position and structural information of the target before being passed into the subsequent two-level mapping structure, it expresses a kind of global semantic information for the target. However, in the steel surface defect detection task, different positions of the steel surface defect image correspond to different defect types, and the detector is very sensitive to the spatial positions of these different defects. Therefore, we expect that the vectors compared in the contrastive learning framework designed for the steel surface defect detection task should retain the position and structural information of various types of steel surface defects as much as possible, rather than using global semantic features for a single steel surface defect.

[0107] The original SimSiam uses a one-dimensional vector obtained after global pooling (AVG POOL). Unlike the original representation model, we retain the convolutional feature map of the steel surface defect image and embed it as a defect image passed into the subsequent mapping structure to achieve the purpose of retaining the structural information of the steel surface defects. The specific approach is: remove the global pooling layer in the backbone feature extraction network ResNet-50 in the original SimSiam, retain the obtained feature map, and continue to pass it into the subsequent nonlinear mapping module in the contrast learning framework. Since the incoming data is changed from the original one-dimensional vector to a feature map, the subsequent MLP structure also needs to be modified accordingly. At the same time, in order to solve the problem that different positions may correspond to different defect types, we use the EMD distance with a matching mechanism to measure the similarity of various types of steel defects, instead of using the cosine similarity of the original model.

[0108] like Figure 2 As shown in the figure, the encoder f without the pooling layer is used to extract features from the original unlabeled steel surface defect data, and the output is a feature map, represented as Y. Since the data passed into the subsequent projection head g is no longer a vector but a feature map, a 1x1 convolution structure is used instead of the original MLP, and the feature map is mapped using the convolution module, so that the new model SimSE maintains the basic structure of the original SimSiam and retains the location and structural information of steel surface defects.

[0109] The specific steps for constructing the upstream steel surface defect characterization learning model are as follows:

[0110] (1) Set the backbone feature extraction network ResNet-50 in SimSiam:

[0111] The global pooling layer in the backbone feature extraction network ResNet-50 in the original SimSiam is removed, and the obtained feature map is retained and continued to be passed into the subsequent nonlinear mapping module in the contrastive learning framework; the encoder f without the pooling layer is used to extract features from the original unlabeled steel surface defect data, and the output is a feature map, denoted as Y.

[0112] (2) A 1x1 convolutional structure is used to replace the original MLP, the convolutional structure is used to map the feature map, and the EMD distance with a matching mechanism is used to measure the similarity of various steel defects.

[0113] Here, the global pooling layer in the backbone feature extraction network ResNet-50 in the original SimSiam is removed, and the obtained feature map is retained and continued to be passed to the subsequent nonlinear mapping module in the contrastive learning framework. Since the incoming data has changed from the original one-dimensional vector to a feature map, the subsequent MLP structure also needs to be modified accordingly. At the same time, in order to solve the problem that different positions may correspond to different defect types, the EMD distance with a matching mechanism is used to measure the similarity of various types of steel defects instead of the cosine similarity of the original model. The encoder f without the pooling layer is used to extract features from the original unlabeled steel surface defect data, and the output is a feature map, represented as Y0. Since the data passed into the subsequent projection head g is no longer a vector but a feature map, we use a 1x1 convolution structure instead of the original MLP, and use the convolution module to map the feature map, so that the new model SimSE maintains the basic structure of the original SimSiam and retains the position and structural information of the steel surface defects.

[0114] The third step is to train the upstream steel surface defect characterization learning model: use the pre-processed unlabeled industrial production steel images to train the upstream steel surface defect characterization learning model.

[0115] The specific steps are as follows:

[0116] (1) Perform two random augmentations on the preprocessed unlabeled industrial steel image x to obtain two augmented views x1 and x2:

[0117] x1=Augmentation random (x),x2=Augmentation random (x);

[0118] (2) The two augmented views x1 and x2 are respectively input into the encoder f for feature extraction to obtain two sets of feature maps Y1 and Y2; the encoder f is a ResNet-50 network that discards the fully connected layer and the global pooling layer, and the encoder f shares weights between the two augmented views;

[0119] (3) Use a 1x1 convolution with a sliding step size of 1 to further map the two sets of feature maps Y1 and Y2, and keep the size of the feature maps unchanged;

[0120] (4) Add one more level of MLP mapping h on one side:

[0121] Considering the computational complexity, a pooling layer is added before this to pool the feature map into a one-dimensional vector for the final similarity calculation. The two vectors are represented as

[0122] and

[0123] Among them, f is the backbone feature extraction network, g is the first-level mapping module, which is composed of 1x1 convolution, and h is the second-level mapping module, which is composed of MLP; P1 is the feature vector obtained after x1 passes through the feature extraction network f and the subsequent two mapping modules g and h, and Z2 is the feature vector obtained after x2 passes through the feature extraction network f and the subsequent mapping module g;

[0124] (5) Use EMD distance to calculate the similarity of two vectors:

[0125] First, the two augmented views are passed through the feature extraction network and the mapping module to obtain two vectors, which are represented as P1 = {x i |i=1,2,3,...m},Z2={y j |j=1,2,3,...n}, the distance between the individual components of the vector is defined as: d(x i ,y j )=|x i -y i |;

[0126] Secondly, define a coupling matrix S, which represents the minimum matching solution between the components of vectors P1 and Z2. In addition, define a distance matrix C, which is used to record the distance from each component of vector P1 to each component of vector Z2. That is, each element C of the matrix ij Respectively represent the distance from the i-th vector of P1 to the j-th vector of Z2. At this time, the EMD distance between the two vectors can be calculated using the Frobenius inner product between the above scheme matrix S and the distance matrix C:

[0127]

[0128] The goal is then transformed into seeking an optimal matching solution that minimizes the total distance. The optimization problem is defined as:

[0129]

[0130] stS1=P1

[0131] S T 1=Z2.

[0132] Since the vectors to be compared have high dimensions, we use the fast iterative method Sinkhorn-Knopp to solve the constrained optimization problem. The algorithm defines the entropy of a matrix as By introducing the regularization coefficient ε, a smoother coupling matrix can be obtained:

[0133]

[0134] stS1=P1

[0135] P T 1=Z2

[0136] By increasing the regularization coefficient ε, the resulting coupling matrix will be smoother. When ε approaches zero, the coupling matrix will be sparser, and the final solution will be closer to the original optimal matching distance problem. At this point, after introducing this entropy regularization, the problem is converted into a convex optimization problem, which can then be solved using the Sinkhorn-Knopp iteration. The optimal matching solution is further expressed as:

[0137] S=diag(u)Kdiag(v)

[0138] During the iteration, the sequences u and v are updated alternately:

[0139]

[0140] Among them, K is the kernel matrix, and its elements are calculated by the distance matrix C, expressed as:

[0141] The iterative algorithm is used to find the minimum match between the components of the two sets of vectors, and the optimal matching matrix S is obtained. Then, the minimum cost of transforming vector P1 into vector Z2 is obtained by calculating the Frobenius inner product of the coupling matrix S and the cost matrix C. The minimum cost is the EMD distance of the two vectors, which is expressed as:

[0142] D EMD (P1,Z2)=<C,S> .

[0143] D EMD (P1,Z2)=||P1-Z2|| EMD .

[0144] At the same time, a symmetric loss is used, that is, the results of exchanging x1 and x2 through the network are expressed as Z1 and P2, and the EMD is also used to calculate the distance between the two vectors:

[0145] D EMD (P2,Z1)=||P2-Z1|| EMD

[0146] The overall loss function is defined as follows:

[0147]

[0148] Among them, Stopgrad() stops the gradient backpropagation operation.

[0149] The steel surface defect dataset is a multi-target dataset, that is, each image contains multiple or multiple types of defect targets. Contrastive representation learning requires a random augmentation of the input original image at the beginning to obtain the positive view of the current sample. This operation will not cause problems on the single-target classification dataset, but the initial random cropping on the steel surface defect dataset is very likely to introduce noise to the model. The potential prior of contrastive learning is that different views of the same object still express the object, and the different views of the target are obtained through some random augmentations. When performing random cropping on the steel defect detection dataset, it is very likely to obtain different types of defect targets. It is obviously unreasonable to regard the views containing multiple different defect types obtained by random cropping as the same defect object, so the present invention uses the EMD distance with a matching mechanism to measure the contrast loss and perform matching similarity calculations between multiple defect targets. Since we canceled the pooling layer in the original SimSiam backbone network, the feature map was retained in the subsequent structure, and the feature map was not pooled until the final unilateral projection, so the final vector retained the position and structural information of the target to a great extent, and the different components of the multi-target defect image vector can also express the information of different objects. The present invention uses the EMD distance to perform the minimum match on the components of the vector, and then calculates the total distance.

[0150] The fourth step is to construct a downstream steel surface defect detector: Based on the backbone feature extraction network obtained by characterization training using a large number of unlabeled images, the downstream steel surface defect detector is constructed.

[0151] After the improved representation learning model SimSE obtains a good representation of the steel surface defect data, it is transferred to the downstream target detector, and then a small amount of labeled steel surface defect data is used to train the steel surface defect detector to obtain a steel surface defect detector with excellent performance, such as Figure 4 Considering that the backbone model used in our representation network is Resnet-50, in order to adapt to the subsequent weight migration, the present invention adopts the two-stage target detector Faster R-CNN with Resnet-50 as the feature extraction network, as shown in Figure 3 The specific steps are as follows:

[0152] (1) For the upstream steel surface defect characterization learning model, the mapping comparison module of the characterization model SimSE is discarded to obtain the backbone feature extraction network ResNet-50.

[0153] (2) Design a weight migration scheme: The weights of the backbone feature extraction network part obtained by SimSE using contrastive representation learning on a large number of unlabeled steel surface defect datasets are loaded into the backbone feature extraction network ResNet-50 of the downstream steel surface defect detector Faster R-CNN by changing the weight name.

[0154] Since the weight parameters of deep learning models are saved in the form of key-value pairs, even if different models contain the same backbone network, if the key names are different, it is difficult to load weights into each other. To this end, the present invention designs a weight migration scheme. We use the method of changing the weight key name to perform weight matching migration of two models.

[0155] A1) Use Amodel_dict to load the weights of the upstream steel surface defect characterization learning model, and use Bmodel_dict to load the weights of the downstream steel surface defect detector;

[0156] A2) traverse the key-value pairs of the weights of the downstream steel surface defect detector, find the key name of the part to be loaded, replace the key name of the part of the upstream steel surface defect characterization learning model with the key name of the part of the downstream steel surface defect detector, and use key_new to receive the changed key value;

[0157] At this time, Amodel_dict[key_new] is the part of the upstream steel surface defect characterization learning model that matches the downstream steel surface defect detector, that is, the same backbone feature extraction network ResNet-50 part;

[0158] A3) Using the weights of the corresponding part of the upstream steel surface defect characterization learning model to overwrite the weights of the corresponding part in the downstream steel surface defect detector.

[0159] The following is the algorithm for the above weight loading method:

[0160] Amodel_dict=load(A_dict)

[0161] Bmodel_dict=load(B_dict)

[0162] for key,value in Bmodel_dict:

[0163] if B.keyname in key:

[0164] key_new=key.replace(B.keyname,A.keyname)

[0165] Bmodel_dict[key]=Amodel_dict[key_new].

[0166] (3) The Faster R-CNN with the steel surface defect characterization weights loaded is used as the downstream steel surface defect detector.

[0167] Step 5: Training of downstream steel surface defect detector: Use pre-processed labeled industrial production steel images to train downstream steel surface defect detector.

[0168] The training of the downstream steel surface defect detector is to extract features from the input image through the backbone network to obtain the feature map of the image; then the feature map is input into the RPN proposal network to generate region proposals and obtain the proposal box; the ROI pooling layer generates the proposal feature map in the feature map output by the feature extraction network according to the region proposal of the RPN network; and then the generated proposal feature map is passed into the subsequent target category determination network and bounding box regression network to obtain the final target category and the precise position of the detection box. The specific steps are as follows:

[0169] (1) The input steel surface defect image is resized to 600x800 and passed into the backbone feature extraction network (ResNet-50) for feature extraction to obtain a set of 38x50 shared feature layers.

[0170] (2) The shared feature layer is passed into the RPN proposal frame network to generate the proposal frame for steel surface defect detection. The shared feature layer is convolved with 3x3, and the result is convolved with 1x1 twice. One of the 1x1 convolution channels has 18, and the result is used to determine whether the prior frame actually contains an object. The other 1x1 convolution channel has 36 channels, which is used to determine the four parameters of the proposal frame, namely the x-axis and y-axis of the center point, and the height and width of the frame. After these two sets of convolutions, the prior frame is adjusted to the proposal frame.

[0171] (3) Combine the proposed box obtained by the RPN network with the original shared feature layer and pass it to the RoIAlign layer to intercept the shared feature layer. Since the size of the proposed box obtained by the previous RPN proposal network is not consistent, the size of the local feature layer we intercept here will also be different. The RoIAlign layer will perform regional pooling on all local feature layers to obtain local feature layers of the same size.

[0172] (4) All local feature layers are passed to the subsequent classification and regression network for classification prediction and regression prediction respectively. The results of the regression prediction network will fine-tune the previously obtained suggestion box to obtain an accurate final detection box. The classification prediction network uses the softmax function for mapping to obtain the probability value of multiple classifications and determine the specific type of steel surface defects in the detection box. The loss function is used to compare the obtained results with the true value, and the gradient descent method is used to adjust the network parameters to gradually reduce the value of the loss function, and finally the trained downstream steel surface defect detector is obtained.

[0173] Step 6: Acquisition of images of industrially produced steel to be tested: Acquisition of images of industrially produced steel to be tested and preprocessing.

[0174] Step 7: Obtaining the defect detection results of industrial steel images: Input the preprocessed industrial steel images to be detected into the trained downstream steel surface defect detector to obtain the defect detection results of the industrial steel images.

[0175] The method used in the present invention does not require a large amount of labeled data, but utilizes unlabeled data, which can greatly reduce the excessive reliance on labeled data in the training of steel defect detectors while ensuring the performance of the detector.

[0176] In the steel surface defect detection task based on visual technology, it is necessary not only to classify the types of various defects in the picture, but also to accurately locate the defect target in the picture. However, the current contrastive learning methods are designed for the representation of classification tasks and are not suitable for steel defect detection tasks. Therefore, we make improvements on the original contrastive learning model (SimSiam) for the steel defect detection task. Specifically, the original contrastive learning model uses the one-dimensional vector obtained after pooling the feature map as the embedding vector of the steel surface defect in the feature space, but this operation will lose the context information of the steel defect target. Unlike the defect classification task, the focus of the defect detection task is to locate and classify the target, and the structure and position information of the defect target are very important. Therefore, we remove the global pooling layer at the end of the original model feature extraction network ResNet, so that the data passed into the subsequent mapping structure is transformed from the one-dimensional vector that does not contain the context information of the defect target to a feature map containing the position and structure information of various steel defect targets. At the same time, the subsequent MLP mapping also needs to be adjusted accordingly. In order to retain the architecture of the original contrast model, 1x1 convolution is used instead of MLP to map the feature map.

[0177] Considering that in the steel defect detection task, the original image generally contains multiple types of steel defects or multiple defect targets. For example, a steel surface defect image may contain multiple types of defects such as welds, punching holes, scratches, etc., and the target task of contrastive learning is to maximize the similarity between the vector representations of the same target in different views to obtain a good representation of the target. If multiple defect types in the steel surface defect image are regarded as the same type of target, it is obviously very unreasonable. Therefore, we introduce the EMD metric to calculate the contrast loss. The matching mechanism of the EMD metric is used to perform local matching similarity calculations on multiple and multi-class targets in the steel surface defect data, eliminating the noise effect generated by multiple targets during the final vector comparison.

[0178] The present invention uses our improved method (SimSE) to perform self-supervised contrastive representation learning on a large-scale unlabeled dataset of steel defects. After obtaining a good representation of the steel defect data, the model is migrated to the target detector Faster R-CNN, and a small number of labeled public steel surface defect datasets NEU-DET and GC10-DET are used to train downstream defect detection tasks. The experimental results show that our steel surface defect detection technology based on self-supervised contrastive representation can achieve very good defect detection results without relying on a large amount of labeled defect data. The designed downstream steel defect detector can process and detect defect images, as well as real-time video streams collected by cameras on industrial production lines.

[0179] SimSiam-FR means using the original characterization model method, and SimSE-FR means using the improved method.

[0180] Table 1 shows the experimental results on the public dataset NEU-DET. The NEU-DET steel surface defect dataset contains six categories, namely cracks (Cr), inclusions (In), patches (Pa), pitting surfaces (Ps), rolling scale (Rs), and scratches (Sc).

[0181] Table 1. Comparison of defect detection accuracy on the NEU-DET dataset

[0182]

[0183] Table 2 shows the experimental results on the public dataset GC10-DET. The GC10-DET steel surface defect dataset contains ten categories, namely, punching (Pu), weld (Wl), crescent-shaped gap (Cg), water spot (Ws), oil spot (Os), silk spot (Ss), inclusion (In), rolling pit (Rp), crease (Cr), and waist fold (Wf).

[0184] Table 2. Comparison of defect detection accuracy on the GC10-DET dataset

[0185]

[0186]

[0187] The above experimental results verify the effectiveness of the steel defect detection technology based on self-supervised contrastive representation learning designed by the present invention. Due to the strong dependence of the steel surface defect detection task on the position and structural information of the defect target, the method of retaining the feature map as the subsequent defect image embedding in the present invention enables the model to fully capture the contextual information of various types of defects on the steel surface. At the same time, the EMD metric adopted by the present invention also eliminates the negative impact on contrastive representation learning caused by the inclusion of multiple types of targets in the obtained pictures when performing random cropping on a multi-target steel surface defect dataset. The steel surface defect detector designed in this way can make full use of the position and structural information of various defects in the original unlabeled data, and can obtain a steel surface defect detector with excellent performance without relying on a large amount of labeled data.

[0188] The method of the present invention is compared with other target detection methods such as SSD, YOLO-V2, YOLO-V4, RetinaNet, and Faster R-CNN, as shown in Table 3.

[0189] Table 3. Performance comparison with other existing technology methods

[0190]

[0191] Note: Input-size in the table is the size of the input image; mAP is the average precision; FPS is the frame rate of the steel detector; Parameter is the parameter volume size of the detection model. The detection speed tests of all models are performed on the GeForce RTX3060Laptop GPU.

[0192] like Figure 5 and Figure 6 As shown, Figure 5 This is the defect detection effect diagram of the NEU-DET dataset. Figure 5 (a) is the detection of crack (Cr) defects. This type of defect is not easy to detect due to its small size and unclear features. However, our technology uses an improved self-supervised contrastive representation learning method to make full use of the information provided by the data itself and uses vectors containing a large amount of contextual information for contrastive learning, which improves the detection of this defect type in downstream detection tasks by 5.5%. In addition, Figure 5 (b) is the detection effect diagram of inclusions (In). Figure 5 (c) is plaque (Pa), Figure 5 (d) is the pitting surface (Ps), Figure 5 (e) is the mill scale (Rs), and Figure 5 (f) is the effect diagram of scratch (Sc) detection. Among them, our technology has the greatest improvement in the detection effect of inclusions (In), achieving an improvement of 11%.

[0193] Figure 6 This is the defect detection effect diagram of the GC10-DET dataset. Figure 6 (a) shows the defect image detection effect including weld (Wl), crescent-shaped gap (Cg), and oil spot (OS). Figure 6 (b) is a diagram showing the inspection results of punching holes (Pu), welds (Wl), and oil spots (Os). Figure 6 (c) is the detection effect diagram of oil spot (Os), Figure 6 (d) is a diagram showing the detection results of three types of defects, namely, punching (Pu), welding (Wl) and rolling pit (Rp). Our technology has improved by six points in the detection of three types of defects, namely, silk spots (Ss), inclusions (In) and waist folds (Wf). In addition, creases (Cr) are extremely difficult to detect, and our technology has achieved a 25.1% improvement in this type of defect.

[0194] This shows the effectiveness of the steel surface defect detection technology we proposed based on self-supervised contrast representation learning.

[0195] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for detecting steel image defects in industrial production based on self-supervised contrastive representation learning technology, It is characterized in that The following steps are involved: 11) Acquisition and preprocessing of industrial steel images: Acquisition of labeled and unlabeled industrial steel images and preprocessing; 12) Construction of upstream steel surface defect characterization learning model: Construction of upstream steel surface defect characterization learning model; The construction of the upstream steel surface defect characterization learning model includes the following steps: 121) Set the backbone feature extraction network ResNet-50 in SimSiam: The global pooling layer in the backbone feature extraction network ResNet-50 in the original SimSiam is removed, and the obtained feature map is retained and continuously passed to the subsequent nonlinear mapping module in the contrastive learning framework; the encoder f without the pooling layer is used to extract features from the original unlabeled steel surface defect data, and the output is a feature map, denoted as Y; 122) A 1x1 convolutional structure is used to replace the original MLP, the convolutional structure is used to map the feature map, and the EMD distance with a matching mechanism is used to measure the similarity of various steel defects; 13) Training of upstream steel surface defect characterization learning model: Using pre-processed unlabeled industrial production steel images to train the upstream steel surface defect characterization learning model; The training of the upstream steel surface defect characterization learning model includes the following steps: 131) Perform two random augmentations on the preprocessed unlabeled industrial steel image x to obtain two augmented views x 1 and x 2 : x 1 =Augmentation random (x),x 2 =Augmentation random (x); 132) The two augmented views x 1 and x 2 Input them into encoder f for feature extraction, and get two sets of feature maps Y 1 and Y 2 ; Encoder f is a ResNet-50 network that discards the fully connected layer and the global pooling layer. Encoder f shares weights between the two augmented views; 133) Two sets of feature maps Y obtained by using 1x1 convolution with a sliding step size of 1 1 and Y 2 Perform further feature mapping and keep the feature map size unchanged; 134) Add one more level of MLP mapping h on one side; Considering the computational complexity, a pooling layer is added before this to pool the feature map into a one-dimensional vector for the final similarity calculation. The two vectors are represented as: P 1 @h(g(f(x 1 )) pooling ) and Z 2 @g(f(x 2 )) pooling , Among them, f is the backbone feature extraction network, g is the first-level mapping module, which is composed of 1x1 convolution, and h is the second-level mapping module, which is composed of MLP; P 1 For x 1 The feature vector Z is obtained after the feature extraction network f and the subsequent two mapping modules g and h. 2 For x 2 The feature vector obtained after passing through the feature extraction network f and a subsequent mapping module g; 135) Use EMD distance to calculate the similarity of two vectors: The loss is symmetric, that is, exchanging x 1 and x 2 The result obtained through the network is expressed as Z 1 and P 2 ; The EMD distance formula and loss function L are defined as follows: D EMD (P,Z 2 )=||P-Z 2 || EMD , Among them, Stopgrad() stops the gradient backpropagation operation; 14) Construction of downstream steel surface defect detector: Based on the backbone feature extraction network obtained by training with a large number of unlabeled images, the downstream steel surface defect detector is constructed; The construction of the downstream steel surface defect detector includes the following steps: 141) For the upstream steel surface defect characterization learning model, the mapping comparison module of the characterization model SimSE is abandoned to obtain the backbone feature extraction network ResNet-50; 142) Design a weight migration scheme: load the weights of the backbone feature extraction network part of SimSE obtained by contrastive representation learning on a large number of unlabeled steel surface defect datasets into the backbone feature extraction network ResNet-50 of the downstream steel surface defect detector Faster R-CNN by changing the weight name; The design weight migration scheme includes the following steps: 1421) Use Amodel_dict to load the weights of the upstream steel surface defect characterization learning model, and use Bmodel_dict to load the weights of the downstream steel surface defect detector; 1422) traverse the key-value pairs of the weights of the downstream steel surface defect detector, find the key name of the part to be loaded, replace the key name of the part of the upstream steel surface defect characterization learning model with the key name of the part of the downstream steel surface defect detector, and use key_new to receive the changed key value; At this time, Amodel_dict[key_new] is the part of the upstream steel surface defect characterization learning model that matches the downstream steel surface defect detector, that is, the same backbone feature extraction network ResNet-50 part; 1423) using the weight of the corresponding part of the upstream steel surface defect characterization learning model to overwrite the weight of the corresponding part in the downstream steel surface defect detector; 143) Using Faster R-CNN loaded with steel surface defect characterization weights as a downstream steel surface defect detector; 15) Training of downstream steel surface defect detector: training downstream steel surface defect detector using pre-processed labeled industrial production steel images; The training of the downstream steel surface defect detector is to extract features from the input image through the backbone network to obtain the feature map of the image; then the feature map is input into the RPN proposal network to generate region proposals to obtain the proposal box; the ROI pooling layer generates the proposal feature map in the feature map output by the feature extraction network according to the region proposal of the RPN network; and then the generated proposal feature map is passed into the subsequent target category determination network and the bounding box regression network to obtain the final target category and the precise position of the detection box; the specific steps are as follows: 151) The input steel surface defect image is adjusted to a size of 600x800, and passed into the backbone feature extraction network ResNet-50 for feature extraction to obtain a set of 38x50 shared feature layers; 152) The shared feature layer is passed into the RPN suggestion frame network to generate the steel surface defect detection suggestion frame; the shared feature layer is convolved with 3x3, and the obtained result is convolved with two 1x1 convolutions, one of which has 18 channels and is used to determine whether the prior frame contains an object. The other 1x1 convolution has 36 channels and is used to determine the four parameters of the suggestion frame, i.e., the x-axis and y-axis of the center point, and the height and width of the frame; after these two sets of convolutions, the prior frame is adjusted to the suggestion frame; 153) The proposed box obtained by the RPN network is combined with the original shared feature layer, and the shared feature layer is intercepted by the RoIAlign layer. The RoIAlign layer performs regional pooling on all local feature layers to obtain local feature layers of the same size; 154) All the local feature layers obtained are passed to the subsequent classification and regression network for classification prediction and regression prediction respectively; the result of the regression prediction network will finely adjust the previously obtained suggestion box to obtain an accurate final detection box; the classification prediction network uses the softmax function for mapping to obtain the probability value of multiple classifications, judge the specific type of steel surface defects in the detection box, use the loss function to compare the obtained results with the true value, use the gradient descent method to adjust the network parameters, gradually reduce the value of the loss function, and finally obtain the trained downstream steel surface defect detector; 16) Acquisition of images of industrial steel to be tested: Acquisition of images of industrial steel to be tested and preprocessing; 17) Obtaining defect detection results of industrial steel images: Input the pre-processed industrial steel images to be detected into the trained downstream steel surface defect detector to obtain defect detection results of the industrial steel images.

2. The method for detecting steel image defects in industrial production based on self-supervised contrastive representation learning technology according to claim 1, It is characterized in that The acquisition and preprocessing of the industrial production steel pictures include the following steps: 21) Obtain pictures of industrially produced steel; 22) Data enhancement processing of industrial steel pictures: First, all industrial steel pictures were resized to 224x224, and then random cropping and random horizontal flipping were applied; then brightness color enhancement with an intensity of 0.4, contrast enhancement with an intensity of 0.4, hue enhancement with an intensity of 0.1, application probability with an intensity of 0.8, and random grayscale transformation with an application probability of 0.2 were applied; 23) Perform Gaussian blur processing on the industrial production steel images after data enhancement.

3. The method for detecting steel image defects in industrial production based on self-supervised contrastive representation learning technology according to claim 1, It is characterized in that The method of calculating the similarity of two vectors using the EMD distance comprises the following steps: 31) The two vectors obtained after the two augmented views pass through the feature extraction network and the mapping module are represented as P 1 ={x i |i=1,2,3,...m},Z 2 ={y j |j=1,2,3,...n}, the distance between the individual components of the vector is defined as: d(x i ,y j )=|x i -y i |, where x and y are vectors P respectively. 1 and Z 2 The various components of 32) Define a coupling matrix S, representing the vector P 1 and Z 2 The minimum matching solution between the components of , and then define a distance matrix C to record the vector P 1 Each component to vector Z 2 The distance between each component, that is, each element C of the matrix ij Respectively represent P 1 The i-th vector to Z 2 The distance of the jth vector; The EMD distance between two vectors is calculated using the Frobenius inner product between the solution matrix S and the distance matrix C. The calculation formula is as follows: 33) Solve the optimal matching solution to minimize the total distance. The optimization problem is defined as: s.t.S1=P 1 S Τ 1=Z 2 ; 34) The fast iterative method Sinkhorn-Knopp is used to solve the constrained optimization problem. The entropy of a matrix is ​​defined as The regularization coefficient ε is introduced to obtain a smoother coupling matrix: s.t.S1=P 1 P Τ 1=Z 2 ; By increasing the regularization coefficient ε, the resulting coupling matrix is ​​smoother; When ε approaches zero, the coupling matrix becomes more sparse, and the final solution is closer to the original optimal matching distance problem; After introducing this entropy regularization, the problem is transformed into a convex optimization problem, which is then solved using the Sinkhorn-Knopp iteration. The optimal matching solution is further expressed as: S = diag(u)Kdiag(v), During the iteration, the sequences u and v are updated alternately: Among them, K is the kernel matrix, and its elements are calculated by the distance matrix C, expressed as: 35) Use the iterative algorithm to find the minimum match between the components of the two sets of vectors, and obtain the optimal matching matrix S. Then, by calculating the Frobenius inner product of the coupling matrix S and the cost matrix C, the vector P is obtained. 1 Convert to vector Z 2 The minimum cost is the EMD distance of the two vectors, expressed as: D EMD (P 1 ,Z 2 )=<C,S>, That is, D EMD (P 1 , Z 2 ) = ||P 1 -Z 2 || EMD ; 36) Use symmetric loss, that is, swap x 1 and x 2 The result obtained through the network is expressed as Z 1 and P 2 , use EMD to calculate the distance between two vectors: D EMD (P 2 ,Z 1 )=||P 2 -Z 1 || EMD , The overall loss function is defined as follows: Among them, Stopgrad() stops the gradient backpropagation operation.

Citation Information

Patent Citations

  • Accelerating semiconductor-related computations using learning based models

    US20170200260A1

  • Liquor bottle surface defect inspection method, electronic device, and storage medium

    WO2021232613A1