Tire flaw detection domain adaptive method based on migratable swintransformer

Abstract

The application provides a tire flaw detection domain self-adaptive method based on a migratable Swin Transformer.In the model training stage, after the tire X-ray image is cropped, it is input into the migratable Swin Transformer feature extractor, then the domain discriminator is used for region-level alignment, and the region-based attention weight is provided for sub-domain self-adaption, then the flaw detection result is obtained through the tire flaw detection classifier, and the model is trained by using the stochastic gradient descent method; in the flaw detection stage, the image obtained by the target X-ray machine is first cropped, and then input into the feature extractor and the tire flaw detection classifier obtained in the model training stage, so that the flaw detection result is obtained.The application can solve the field deviation problem existing in the tire flaw detection and improve the flaw detection efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of machine vision automatic detection, and in particular to a tire flaw detection domain adaptive method based on a transferable SwinTransformer. BACKGROUND

[0002] In the production process of tires, various defects inevitably occur, so in order to ensure the safety and reliability of tire application, quality inspection must be carried out before the tire is shipped. With the improvement of factory automation level, in recent years, many computer vision-based methods have been widely used in tire flaw detection, and in particular, due to the development of computer technology and industrial big data, data-driven models based on deep learning have received extensive attention in flaw detection. Compared with traditional flaw detection methods, deep learning-based methods can achieve satisfactory performance when there is sufficient labeled data. However, due to the scarcity of tire flaws in actual industrial sites, it is difficult to obtain the required amount of data for deep learning models. Moreover, the performance of deep learning models can only be guaranteed when the source domain and the target domain conform to the same distribution, which is impractical in most cases. The working conditions in the tire production process are complex and changeable, and changes in working conditions will result in distribution differences between the source domain and the target domain (i.e. domain shift), thereby causing a significant decrease in the performance of deep learning-based defect detection models. At this time, it is necessary to collect and label a large number of target domain samples, which is very time-consuming and inefficient. Moreover, tire X-ray images themselves have low brightness and low contrast characteristics, and there are problems such as large variation in flaw size and unfixed appearance position. The above characteristics have caused difficulties in industrial tire flaw detection. SUMMARY

[0003] The present application aims at the deficiencies of the prior art, and provides a tire flaw detection domain adaptive method based on a transferable SwinTransformer.

[0004] To this end, the present application adopts the following technical solutions:

[0005] A tire flaw detection domain adaptive method based on a transferable Swin Transformer, characterized in that: the tire flaw detection method uses a Swin Transformer as an image feature extractor, simultaneously considering fine-grained alignment and semantic information alignment of different position regions of the image, thereby effectively extracting and learning transferable and discriminative features in tire flaw detection, and only a small amount of unlabeled target domain samples are required to enable the model trained on the source domain to also achieve excellent performance on the target domain, effectively solving the problem of domain shift in tire flaw detection; the tire flaw detection method includes model training and flaw detection.

[0006] The model training includes the following steps:

[0007] Step S1, Data Acquisition: The source X-ray machine on the tire quality inspection line scans the tires that need to be inspected to obtain initial tire X-ray images. The initial tire X-ray images are then cropped into samples of a specified size. The source domain samples are the n samples obtained from the source X-ray machine. s Zhang has labeled images; the target domain samples are obtained from sampling on the target X-ray machine. t Zhang unlabeled image;

[0008] Step S2, Feature extraction: Input b source domain samples and b target domain samples into the feature extractor, which is a Swing Transformer;

[0009] Step S3, Region Alignment: In the Swing TransformerBlock, the last stage of the feature extractor in step S2, a domain discriminator is added to W-MSA and SW-MSA to perform region-level alignment and obtain the relative importance of each region in the image.

[0010] Step S4, Subdomain Adaptation: Based on the final features obtained by the feature extractor in Step S2 and the relative importance of each region of the image obtained in Step S3, fine-grained subdomain adaptation is performed using the local maximum mean difference.

[0011] Step S5, Defect Classification: Input the adapted source domain sample features and target domain sample features from step S4 into the tire defect detection classifier to obtain the defect classification result;

[0012] Step S6, update model parameters: Repeat steps S2 to S5 to update model parameters using stochastic gradient descent, with I iterations;

[0013] Step S7, Save model parameters: After updating the model parameters by I iterations according to step S6, save the model parameters of the feature extractor in step S2 and the tire defect detection classifier in step S5.

[0014] The defect detection includes the following steps:

[0015] Step Z1, Data Preprocessing: Input the target domain tire into the target X-ray machine, and crop the image obtained by the target X-ray machine to obtain a test dataset of a specified size;

[0016] Step Z2, Feature Extraction: Input the test dataset obtained in step Z1 into the feature extractor saved in step S7 of the model training phase to obtain domain-invariant feature representations;

[0017] Step Z3, Defect Detection: Input the domain-invariant feature representation obtained in step Z2 into the tire defect detection classifier saved in step S7 of the model training stage to perform defect detection, and obtain the defect detection results on the test dataset.

[0018] While adopting the above technical solutions, the present invention may also adopt or combine the following further technical solutions:

[0019] As a preferred embodiment of the present invention: the specified size is 224×224 resolution; in step S1, the number of source domain samples n s Preferably, there are no fewer than 100 images in each category, and they need to be labeled. The number of samples in the target domain is n. t Ideally, there should be no fewer than 20 images in each category, and no labeling is required.

[0020] As a preferred technical solution of the present invention: in step S2, the parameter b is preferably 32, and the initialization parameters of the feature extractor Swing Transformer are preferably pre-trained using the ImageNet dataset.

[0021] As a preferred technical solution of the present invention: in step S3, the relative importance of each region of the image is preferably measured using entropy.

[0022] As a preferred technical solution of the present invention: in step S4, the subdomain adaptation requires the use of sample category labels, wherein the category labels of the source domain samples are derived from the annotation information, and the category labels of the target domain samples are derived from pseudo-labels.

[0023] As a preferred technical solution of the present invention: in step S5, the tire defect detection classifier is preferably a two-layer fully connected layer, and adopts a form of source domain and target domain sharing.

[0024] As a preferred technical solution of the present invention: in step S6, the number of iterations I is preferably 5000.

[0025] As a preferred technical solution of the present invention: in step S7, the parameters of the feature extractor model are saved for obtaining the domain-invariant feature representation in step Z2 of the defect detection stage, and the parameters of the tire defect detection classifier model are saved for defect detection in step Z3 of the defect detection stage.

[0026] According to the purpose of this invention, this invention provides an adaptive method for tire defect detection domain based on a transferable Swing Transformer, characterized in that it is trained using the model training method described above.

[0027] According to the purpose of this invention, a non-transitory computer-readable storage medium is provided thereon storing a computer program, characterized in that the computer program, when executed by a processor, implements the above-mentioned model training steps and defect detection steps.

[0028] According to the purpose of this invention, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the above-described model training steps and defect detection steps.

[0029] The adaptive method for tire defect detection domain based on transferable Swin Transformer provided in this invention has the following advantages:

[0030] (1) It can effectively alleviate the problem of domain shift in tire defect detection under different X-ray machines. Only a small number of unlabeled target domain samples are needed to achieve excellent migration effect and improve the efficiency of tire defect detection.

[0031] (2) Utilizing the inherent characteristics of the Swing Transformer, fine-grained alignment based on the self-attention mechanism is performed based on the region, and the region-level features are weighted to improve the accuracy of subdomain adaptation.

[0032] (3) Combining region-based local alignment with subdomain adaptation based on semantic category information effectively extracts and learns transferable and discriminative features. Attached Figure Description

[0033] Figure 1 This is a flowchart of the main process of the method of the present invention;

[0034] Figure 2 This is a diagram of the portable Swing Transformer Block architecture for the method of this invention;

[0035] Figure 3 This is a flowchart of the main process of the model training phase of the method of the present invention;

[0036] Figure 4 This is a flowchart of the main process of the defect detection stage of the method of the present invention. Detailed Implementation

[0037] The specific embodiments of the present invention will be described in detail with reference to the accompanying drawings and specific examples of tire defect detection under field offset conditions.

[0038] In this specific embodiment, five types of defects—thin tires, bent cords, overlapping cords, foreign matter, and bubbles—are selected, along with six categories of normal samples, for tire defect detection under neighborhood shift conditions. Mutual transfer between three different X-ray machine models is used as a specific embodiment, constructing six different transfer tasks. Image preprocessing involves cropping the X-ray machine-sampled images to a 224×224 resolution. The source domain samples include the aforementioned six categories of images, with 100 labeled sample images for each category, sampled from the source X-ray machine. The target domain samples are sampled from the target X-ray machine. The training set in the target domain consists of 20 sample images per category, and the test set consists of 40 sample images per category. The target domain samples are unlabeled images. This specific embodiment is completed by sequentially executing the model training phase and the defect detection phase.

[0039] like Figure 3 As shown, the steps in the model training phase are as follows:

[0040] Step S1: Tire X-ray images are sampled from the source X-ray machine and cropped into 224×224 resolution sample images. A total of 100 images of each class are collected from this X-ray machine and labeled with class tags to form the source domain dataset. Similarly, tire X-ray images are acquired from another target X-ray machine of a different model, cropped into 224×224 resolution sample images, and 20 images of each class are collected as the target domain training set.

[0041] Step S2: Take the 32 source domain samples and 32 target domain samples obtained in step S1 and input them into the feature extractor Swing Transformer.

[0042] Step S3: In the Swing Transformer Block, the last stage of the feature extractor in step S2, a domain discriminator is added to W-MSA and SW-MSA to achieve fine-grained alignment based on regions. Therefore, W-MSA and SW-MSA are renamed W-TMSA and SW-TMSA respectively. The detailed structure is as follows: Figure 2 As shown. Specifically, in order to achieve region-level alignment and assign appropriate attention weights to different image regions, the domain discriminator... Apply to all patches:

[0043]

[0044] Where n = n s +n t P is the number of patches. The loss is cross-entropy, f(·) represents the feature extractor, and y i represent The domain label. To quantify the portability of the patch token, an entropy functional is introduced: Based on this, we will replace MSA with Figure 2 The transferable MSA, or TMSA, is shown in the dashed box:

[0045]

[0046] Where K p Let be the key of the patch token, ⊙ represent the Hadamard product, and [;] represent the join operation. Then we can obtain the expression for S(W)-TMSA:

[0047]

[0048] Therefore, the two consecutive Swing Transformer Blocks in the final stage of the feature extractor can be calculated by the following formula:

[0049]

[0050] in This indicates the output characteristics of the W-TMSA and SW-TMSA modules, z l This represents the output features of the MLP module. This allows regions that are easier to transfer to be assigned greater attention weights, thus revealing the relative importance of each region in the image.

[0051] Step S4: Based on the features finally obtained by the feature extractor in step S2 and the relative importance of each region of the image obtained in step S3, subdomain adaptation is performed using Local Maximum Mean Difference (LMMD):

[0052]

[0053] Where, x s and x t E represents the source domain sample features and the target domain sample features, respectively. c (·) represents the mathematical expectation of category c, p (c) and q( c ) are the sample distributions of source domain class c and target domain class c, respectively. Let Hilbert space be the reproducing kernel defined by the salient kernel, and φ(·) represent the original data to Hilbert space. Mapping. p (c) and q (c) Representing subfields respectively and The distribution of the sample. Assume the weight of each sample belonging to each class is w. c Then the unbiased estimate of equation (5) is:

[0054]

[0055] in and They represent and The weights belonging to category c. Then sample x i The weight can be calculated by the following formula:

[0056]

[0057] Where y ic It is y i The c-th component. Then, according to the kernel trick, an unbiased estimate of the LMMD loss can be obtained:

[0058]

[0059] in and These represent the activation values ​​of the source and target domain sample images after passing through the feature extractor, respectively.

[0060] Step S5: Input the adapted source and target domain sample features from step S4 into the tire defect detection classifier to obtain the defect detection result. The tire defect detection classifier consists of two fully connected layers, and the classification loss in the source domain can be defined as:

[0061]

[0062] in represent The true sample labels are FC, which is a tire defect detection classifier. The objective function of the model consists of the source domain classification loss, the domain discriminator loss, and the local maximum mean difference loss.

[0063]

[0064] Where θ f ,θ c and θ d λ and μ represent the parameters of the feature extractor, tire defect detection classifier, and domain discriminator, respectively. λ and μ represent the balancing parameters. In this specific embodiment, the balancing parameters λ and μ are set to λ = μ = 2 / (1 + exp(-10·t)), where t is the ratio of the current iteration number to the total number of iterations.

[0065] Step S6: Repeat steps S2 to S5, updating the model parameters using stochastic gradient descent for a total of 5000 iterations, employing a dynamic learning rate adjustment strategy. Where k changes linearly from 0 to 1, α0 = 0.01, γ = 10, and β = 0.75.

[0066] Step S7: After completing the number of iterations described in Step S6, save the model parameters of the feature extractor described in Step S2 and the tire defect detection classifier described in Step S5 for use in the defect detection stage.

[0067] like Figure 4 As shown, the steps in the defect detection stage are as follows:

[0068] Step Z1: Pass the tire through the target domain X-ray machine, and then crop the resulting image into 224×224 resolution sample images, obtaining a total of 40 samples for each class as the target domain test set;

[0069] Step Z2: Input the test dataset obtained in step Z1 into the feature extractor saved in step S7 of the model training phase to obtain the domain-invariant feature representation;

[0070] Step Z3: Input the domain-invariant feature representation obtained in step Z2 into the tire defect detection classifier saved in step S7 of the model training stage for defect detection, and obtain the defect detection results on the test dataset.

[0071] Results Verification: To verify the effectiveness of the proposed domain adaptation method for tire defect detection based on the transferable Swin Transformer, this method was compared with CNN and SwinT (SwinTransformer) without transfer learning, as well as existing domain adaptation models BNM, Deep CORAL, DAN, DAAN, DANN, MRAN, and DSAN. Table 1 shows the comparison of tire defect detection accuracy (%) in six different transfer tasks. All experiments were repeated at least three times to enhance the validity of the results. It can be seen that CNN only achieved an average defect detection accuracy of 56.46%, indicating that the defect detection model trained on source domain data cannot be directly applied to the target domain. Furthermore, the superior performance of the domain adaptation method in each transfer task demonstrates its necessity in tire defect detection under domain offset conditions. The average accuracy of the proposed method (TST) is 96.17%, which is at least 8.83% higher than other state-of-the-art domain adaptation methods. In addition, TST(w / p) in Table 1 indicates that the proposed method uses only subdomain adaptation without the transferable Swin TransformerBlock. Compared to TST(w / p), the method of this invention improves the average defect detection accuracy by 3.87%, demonstrating the effectiveness of region alignment-based methods. The above analysis demonstrates the superior performance of the method of this invention.

[0072] Table 1

[0073] Algorithm 1→2 2→1 1→3 3→1 2→3 3→2 Average CNN 51.44±2.27 55.42±2.96 59.00±0.90 59.69±3.69 59.72±1.57 53.47±1.87 56.46 BNM 73.47±1.71 75.14±1.20 79.31±1.75 73.75±0.90 66.80±1.38 69.17±2.23 72.94 SwinT 78.00±1.25 78.54±2.61 79.58±2.62 81.94±1.04 76.15±0.95 76.04±2.74 78.38 Deep CORAL 73.47±1.71 74.58±0.68 82.50±0.68 81.81±1.19 80.28±0.78 80.14±0.52 78.80 DAN 82.64±1.29 85.28±0.85 83.33±1.48 82.36±1.04 79.27±1.03 82.36±0.85 82.54 DAAN 84.72±1.09 86.39±0.40 87.36±1.68 87.64±0.52 83.58±0.56 82.36±0.85 85.34 DANN 87.78±1.90 86.46±1.12 89.17±0.51 87.08±0.90 85.52±1.72 85.21±0.47 86.87 MRAN 86.25±1.36 85.32±1.04 89.72±0.78 87.36±0.20 85.00±1.79 85.14±1.38 86.47 DSAN 88.06±1.68 88.05±1.37 90.00±0.34 88.05±0.71 85.73±0.68 89.86±0.52 88.29 TST(w / p) 93.89±0.71 92.50±0.59 93.19±0.79 92.19±0.74 90.84±1.73 92.92±0.90 92.59 TST 98.05±0.39 94.48±0.45 96.36±0.35 95.73±0.62 95.42±0.34 96.98±0.34 96.17

[0074] From the above description of the embodiments, those skilled in the art will clearly understand that the facilities of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Embodiments of the present invention can be implemented using existing processors, or by dedicated processors used for this or other purposes for suitable systems, or by hardwired systems. Embodiments of the present invention also include non-transitory computer-readable storage media, comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon; such machine-readable media can be any available medium accessible by a general-purpose or special-purpose computer or other machine with a processor. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disc storage, disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of machine-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer or other machine with a processor. When information is transmitted or provided to a machine via a network or other communication connection (hardwired, wireless, or a combination of hardwired and wireless), that connection is also considered a machine-readable medium.

[0075] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A tire flaw detection domain adaptive method based on a migratable Swin Transformer, characterized in that: The method uses Swin Transformer as a feature extractor of an image, while considering fine-grained alignment of different position regions of the image and alignment of semantic information, and the method comprises model training and defect detection; The model training comprises the following steps: Step S1, data acquisition: using a source X-ray machine on a tire quality inspection line to scan a tire that needs to be quality detected, obtaining an initial tire X-ray image, and cutting the initial tire X-ray image to obtain a sample of a specified size; the source domain sample is obtained by sampling on the source X-ray machine n s Zhang has labeled images, and the target domain sample is obtained by sampling on the target X-ray machine n t Zhang has no labeled images; Step S2, feature extraction: input b source domain samples and b target domain samples into a feature extractor, and the feature extractor is Swin Transformer; Step S3, region alignment: in the last stage of the feature extractor in step S2, region-level alignment is performed on W-MSA and SW-MSA in the Swin Transformer Block, and the relative importance of each region of the image is obtained; Step S4, sub-domain adaptation: based on the features finally obtained by the feature extractor in step S2 and the relative importance of each region of the image obtained in step S3, fine-grained sub-domain adaptation is performed using local maximum mean difference; Step S5, defect detection: input the adapted source domain sample features and target domain sample features in step S4 into a tire defect detection classifier to obtain a defect detection result; Step S6, update model parameters: repeat steps S2 to S5, and update the model parameters using the stochastic gradient descent method, and the iteration number is I times; Step S7, save model parameters: after I iteration number updates of the model parameters are completed in step S6, save the model parameters of the feature extractor in step S2 and the tire defect detection classifier in step S5; The defect detection comprises the following steps: Step Z1, data preprocessing: input a target tire into a target X-ray machine, and crop the image scanned by the target X-ray machine to obtain a test data set of a specified size; Step Z2, feature extraction: input the test data set obtained in step Z1 into the feature extractor saved in step S7 of the model training stage to obtain domain-invariant feature representation; Step Z3, defect detection: input the domain-invariant feature representation obtained in step Z2 into the tire defect detection classifier saved in step S7 of the model training stage to perform defect detection, and obtain a defect detection result on the test data set.

2. The method of claim 1, wherein, The size in the step S1 is 224*224 resolution; the number of source domain samples n s No less than 100 for each category, and needs to be labeled; the number of target domain samples n t No less than 20 for each category, and does not need to be labeled.

3. The method of claim 1, wherein the method further comprises: In step S2, the parameter b is 32, and the initialization parameters of the feature extractor Swin Transformer are pre-trained using an ImageNet data set.

4. The method of claim 1, wherein, In step S3, the relative importance of each region of the image is measured by entropy.

5. The method of claim 1, wherein, In step S4, the sub-domain adaptation needs to use the class labels of the samples, wherein the class labels of the source domain samples are derived from the annotation information, and the class labels of the target domain samples are derived from pseudo labels.

6. The method of claim 1, wherein, In step S5, the defect detection classifier is a double-layer fully connected layer, and the defect detection classifier adopts a shared form of the source domain and the target domain.

7. The method of claim 1, wherein, The model saved in step S7 will be applied in the defect detection stage for prediction.

8. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to implement the tire defect detection domain adaptation method in claim 1.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the tire defect detection domain adaptation method in claim 1.

Images