Defect detection method, defect detection system and storage medium

Abstract

The invention provides a defect detection method, a defect detection system and a computer readable storage medium. The defect detection method comprises the following steps: acquiring a target domain sample to be detected; and inputting a target domain sample to be detected into the neural network detection model for defect detection, wherein the step of training the neural network detection model comprises: adding a domain adaptation structure into an initialized basic target detection network to determine a domain adaptation detection network, the domain adaptation structure comprising a feature decoupling domain adaptation module and a bounding box domain adaptation module; inputting a group of source domain images and target domain images into a domain adaptation detection network for training until a network overall loss function reaches a preset target, and obtaining a trained neural network model; and removing a domain adaptation structure in the neural network model to obtain a neural network detection model.

Description

Technical Field

[0001] The present invention relates to the field of defect target detection, and in particular to a defect detection method, a defect detection system, and a computer-readable storage medium. Background Art

[0002] In recent years, the application of unsupervised domain adaptation (UDA) in defect target detection has received increasing attention. Automated and intelligent detection systems play a vital role in the manufacturing industry. However, due to the high cost of data annotation and differences between domains, directly applying a model from one domain (such as a laboratory environment) to another domain (such as an actual production environment) usually results in a significant drop in performance. Therefore, how to achieve efficient defect detection in an unlabeled target domain is an urgent problem to be solved in this field.

[0003] In the task of defect target detection, domain differences are mainly manifested in differences in imaging conditions, object surface characteristics, defect types, etc. For example, high-quality images collected in a laboratory environment and images collected in an actual production environment may have significant differences in lighting, background, noise, etc. These differences will lead to poor generalization performance of the trained model in the target domain.

[0004] Unsupervised domain adaptation aims to train the model by utilizing the features of the source domain (labeled data) and the target domain (unlabeled data) so that it can also perform well in the target domain in actual use. The existing specific methods of unsupervised domain adaptation mainly include the following categories: adversarial training methods, feature mapping methods, and self-supervised learning. Unsupervised domain adaptation technology can better cope with inter-domain differences, so it is introduced into defect target detection technology.

[0005] Adversarial training methods introduce adversarial training mechanisms to align the feature distributions of the source domain and the target domain as much as possible. Typical methods include Domain-Adversarial Neural Networks (DANN) and Adversarial Discriminative Domain Adaptation (ADDA). Feature mapping methods map the data of the source domain and the target domain to the same feature space by learning the common feature space of the source domain and the target domain, thereby reducing the difference between domains. Such methods include CORAL (Correlation Alignment) and Deep CORAL (Deep Correlation Alignment). Self-supervised learning uses unlabeled data in the target domain for self-supervised learning to improve the model's feature extraction capabilities in the target domain.

[0006] In addition, in the task of defect target detection, defects are usually rare events, and they have various shapes and sizes. This makes it difficult for traditional target detection methods to be directly applied. In the prior art, the detection method of Region Proposal Network (RPN), such as Faster R-CNN, is used to solve this problem.

[0007] Existing research on unsupervised domain adaptation defect object detection mainly focuses on the following aspects: adversarial domain alignment, self-supervision and pseudo-labeling, and multi-level feature alignment. Adversarial domain alignment uses adversarial training methods to align the features of the source domain and the target domain as much as possible in the feature space, thereby improving the detection performance of the model in the target domain. Self-supervision and pseudo-labeling generate pseudo-labels in the target domain, and improve the utilization efficiency of the target domain data through self-supervised learning. Multi-level feature alignment not only aligns global features, but also local features and multi-level features to achieve more refined domain adaptation effects. However, the existing technology still faces many challenges, such as the complexity of domain differences, the scarcity and diversity of defect samples, etc.

[0008] In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of a defect detection technology that can improve the domain generalization capability and improve the classification performance and positioning performance of defect detection. Summary of the invention

[0009] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0010] In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a defect detection method, a defect detection system, and a computer-readable storage medium, which can improve the domain generalization ability and improve the classification performance and positioning performance of defect detection.

[0011] Specifically, the above-mentioned defect detection method provided according to the first aspect of the present invention includes the steps of: obtaining a target domain sample to be detected; and inputting the target domain sample to be detected into a neural network detection model for defect detection; wherein the step of training the neural network detection model includes: adding a domain adaptation structure to an initialized basic target detection network to determine a domain adaptation detection network, the domain adaptation structure including a feature decoupling domain adaptation module and a bounding box domain adaptation module; inputting a group of source domain images and target domain images into the domain adaptation detection network for training until the overall loss function of the network reaches a preset target to obtain a trained neural network model; removing the domain adaptation structure in the neural network model to obtain the neural network detection model.

[0012] Preferably, in an embodiment of the present invention, the step of training the neural network detection model further comprises: inputting the source domain data set into the basic object detection network for training to determine the initialized basic object detection network.

[0013] Preferably, in one embodiment of the present invention, the feature decoupling domain adaptation module includes a shallow feature decoupling domain adaptation module and a deep feature decoupling domain adaptation module, and the domain adaptation structure also includes a cascaded RPN domain adaptation module and a classification domain adaptation module.

[0014] Preferably, in one embodiment of the present invention, the initialized basic target detection network includes a detector structure and a classification and regression module, the detector structure includes a feature extraction network module, a region candidate network module and an interest domain alignment module, and the step of adding the domain adaptation structure to the initialized basic target detection network to determine the domain adaptation detection network includes: configuring the shallow feature decoupling domain adaptation module and the deep feature decoupling domain adaptation module in the feature extraction network module, configuring the cascade RPN domain adaptation module in the region candidate network module, and configuring the classification domain adaptation module and the bounding box domain adaptation module in the classification regression module to determine the domain adaptation detection network.

[0015] Preferably, in an embodiment of the present invention, the feature decoupling domain adaptation module includes a first feature extractor for extracting inter-domain instance-specific distribution invariant features and a second feature extractor for extracting inter-domain invariant features.

[0016] Preferably, in one embodiment of the present invention, the shallow feature decoupling domain adaptation module is configured in the shallow feature extraction part of the feature extraction network, and the deep feature decoupling domain adaptation module is configured in the deep feature extraction part of the feature extraction network.

[0017] Preferably, in an embodiment of the present invention, the cascaded RPN domain adaptation module includes two cascaded RPNs, and the feature extractor of the RPN is spliced ​​with the basic domain adaptation module.

[0018] Preferably, in one embodiment of the present invention, the RPN uses adaptive convolution as a feature extraction function.

[0019] Preferably, in an embodiment of the present invention, the classification domain adaptation module and the bounding box domain adaptation module are in a cascade relationship in structure.

[0020] Preferably, in an embodiment of the present invention, the classification domain adaptation module includes a feature extractor, a category identifier and a basic domain adaptation module.

[0021] Preferably, in an embodiment of the present invention, the bounding box domain adaptation module includes a feature extractor, a main regressor and an adversarial regressor.

[0022] Preferably, in one embodiment of the present invention, the overall network loss function includes a first network loss function, a second network loss function and a third network loss function, the first network loss function includes a feature decoupling domain adaptation module loss function, a feature basis domain discrimination loss function and an association consistency loss function, the second network loss function includes a cascaded RPN domain adaptation module loss function, and the third network loss function includes a classification domain adaptation module loss function and a bounding box domain adaptation module loss function.

[0023] Preferably, in one embodiment of the present invention, the feature decoupling domain adaptation module loss function includes a shallow feature decoupling domain adaptation module loss function and a deep feature decoupling domain adaptation module loss function, the feature basic domain discrimination loss function includes a shallow feature basic domain discrimination loss function and a deep feature basic domain discrimination loss function, and the first network loss function also includes a mutual information loss function.

[0024] Preferably, in an embodiment of the present invention, the third network loss function includes a classification regression loss function.

[0025] Preferably, in one embodiment of the present invention, the step of inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall network loss function reaches a preset target to obtain a trained neural network model includes: S1: inputting a set of source domain images and target domain images into the feature extraction network of the domain adaptation detection network to determine instance-specific distribution invariant features, and calculating the feature decoupling domain adaptation module loss function and the feature-based domain discrimination loss function; S2: inputting the instance-specific distribution invariant features into the cascaded RPN domain adaptation module to determine candidate boxes, and calculating the second network loss function; S3: extracting the feature representation corresponding to the instance-specific distribution invariance feature according to the candidate box, and calculating the association consistency loss function; S4: inputting the feature representation into the classification domain adaptation module for domain adaptation adversarial training to determine the feature representation containing foreground information, and calculating the classification domain adaptation module loss function; S5: inputting the feature representation containing foreground information into the bounding box domain adaptation module, performing bounding box domain adaptation adversarial training, and calculating the bounding box domain adaptation module loss function; and S6: repeating steps S1 to S5 until the overall network loss function reaches the preset target.

[0026] Preferably, in one embodiment of the present invention, the step S1 includes: S11: inputting the set of source domain images and target domain images into the shallow feature extraction part of the feature extraction network of the domain adaptation detection network to extract shallow features; S12: inputting the shallow features into the shallow feature decoupling domain adaptation module to extract the first instance-specific distribution invariance features, and calculating the shallow feature decoupling domain adaptation module loss function and the shallow feature basic domain discrimination loss function; S13: adding the shallow features to the first instance-specific distribution invariance features to determine the first features; S14: inputting the first features into the deep feature extraction part of the feature extraction network to extract deep features; S15: inputting the deep features into the deep feature decoupling domain adaptation module to extract the instance-specific distribution invariance features, and calculating the deep feature decoupling domain adaptation module loss function and the deep feature basic domain discrimination loss function.

[0027] Preferably, in one embodiment of the present invention, the overall network loss function also includes a reconstruction loss function, and the step of inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall network loss function reaches a preset target to obtain a trained neural network model also includes a feature reconstruction process, and the step S15 includes: inputting the deep features into the deep feature decoupling domain adaptation module to extract inter-domain invariant features; the feature reconstruction process includes the steps of: according to the candidate box, extracting a first feature representation and a second feature representation corresponding to the deep features and the inter-domain distribution invariance features; performing feature reconstruction on the feature representation and the second feature representation, and calculating the reconstruction loss function based on the first feature representation.

[0028] In addition, the defect detection system provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and is configured to execute the computer instructions stored in the memory to implement the defect detection method provided by any one of the above embodiments.

[0029] In addition, the computer-readable storage medium provided according to the third aspect of the present invention stores computer instructions, which, when executed by a processor, implement the defect detection method provided by any one of the above embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0031] Figure 1 A flow chart of an online analysis phase of a defect detection method provided according to some embodiments of the present invention is shown;

[0032] Figure 2 A flow chart of an offline training phase of a defect detection method provided according to some embodiments of the present invention is shown;

[0033] Figure 3 A schematic diagram of a basic target detection network provided according to a preferred embodiment of the present invention is shown;

[0034] Figure 4 A schematic diagram of a domain adaptation detection network provided according to a preferred embodiment of the present invention is shown;

[0035] Figure 5 A schematic diagram of a feature decoupling domain adaptation module provided according to a preferred embodiment of the present invention is shown;

[0036] Figure 6 A schematic diagram of a cascaded RPN domain adaptation module provided according to a preferred embodiment of the present invention is shown;

[0037] Figure 7 A schematic diagram of a classification domain adaptation module provided according to a preferred embodiment of the present invention is shown;

[0038] Figure 8 A schematic diagram of a bounding box domain adaptation module provided according to a preferred embodiment of the present invention is shown;

[0039] Fig. 9 A flowchart of training a domain adaptation detection network according to some embodiments of the present invention is shown;

[0040] Fig.10 The results show the accuracy of defect detection on target domain data without label information by the defect detection method provided by the preferred embodiment of the present invention; and

[0041] Fig.11 A schematic diagram of a defect detection system provided according to some embodiments of the present invention is shown.

[0042] Reference numerals:

[0043] 100: Defect detection method;

[0044] 300: Basic object detection network;

[0045] 310: feature extraction network module;

[0046] 320: region candidate network module;

[0047] 330: interest domain alignment module;

[0048] 340: Classification and regression module;

[0049] 400: Domain Adaptation Detection Network;

[0050] 411: Shallow Feature Decoupling Domain Adaptation Module;

[0051] 412: Deep Feature Decoupling Domain Adaptation Module;

[0052] 413: Basic domain adaptation module;

[0053] 414: basic domain adaptation module;

[0054] 420: Cascade RPN domain adaptation module;

[0055] 430: classification domain adaptation module;

[0056] 440: Bounding Box Domain Adaptation Module;

[0057] 510: basic domain adaptation module;

[0058] 610: feature extractor;

[0059] 620: convolutional layer;

[0060] 621: candidate box;

[0061] 622: Target score;

[0062] 630: basic domain adaptation module;

[0063] 640: candidate box;

[0064] 710: basic domain adaptation module;

[0065] 1000: Defect detection method;

[0066] 1001, 1002, 1003, 1004: model;

[0067] 1100: Defect detection system;

[0068] 1110: memory;

[0069] 1111: Computer readable storage medium;

[0070] 1120: Processor;

[0071] S110~S120: steps;

[0072] S210-S230: steps; and

[0073] S1~S5: steps. DETAILED DESCRIPTION

[0074] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0075] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0076] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0077] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0078] As mentioned above, existing research on unsupervised domain adaptation defect object detection mainly focuses on the following aspects: adversarial domain alignment, self-supervision and pseudo-labeling, and multi-level feature alignment. Adversarial domain alignment uses adversarial training methods to align the features of the source domain and the target domain as much as possible in the feature space, thereby improving the detection performance of the model in the target domain. Self-supervision and pseudo-labeling generate pseudo-labels in the target domain, and improve the utilization efficiency of target domain data through self-supervised learning. Multi-level feature alignment not only aligns global features, but also local features and multi-level features to achieve more refined domain adaptation effects. However, there are still many challenges in the existing technology, such as the complexity of domain differences, the scarcity and diversity of defect samples, etc.

[0079] In order to overcome the above-mentioned defects existing in the prior art, the present invention provides a defect detection method, a defect detection system, and a computer-readable storage medium, which can improve the domain generalization ability and improve the classification performance and positioning performance of defect detection.

[0080] The prior art mainly explores the distribution between the aligned source domain and the target domain, but ignores the influence of the domain-specific information present in the aligned features. In addition, when transferring detection capabilities across domains, it is very important to extract instance-level features with domain invariance. The present invention improves the classification performance by using a feature decoupling domain adaptation mechanism to decompose inter-domain invariant features and inter-domain instance invariant features. In addition, the prior art mainly focuses on category domain adaptation, while ignoring another important part of target detection, namely, domain adaptation for bounding box regression. The present invention can effectively improve the positioning performance by training a decoupled classification domain adaptation module and a bounding box domain adaptation module. More preferably, the present invention can also adopt a cascaded RPN domain adaptation module to improve the accuracy of candidate box extraction while paying attention to inter-domain consistency features.

[0081] In some non-limiting embodiments, the defect detection method provided by the first aspect of the present invention may be implemented via the defect detection system provided by the second aspect of the present invention.

[0082] The following will first describe the working principle of the above-mentioned defect detection system in conjunction with some examples of defect detection methods. Those skilled in the art will understand that these examples of defect detection methods are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the defect detection system. Similarly, the defect detection system is also only a non-limiting implementation method provided by the present invention, and does not constitute a limitation on the execution subject and execution order of each step in these defect detection methods.

[0083] In some embodiments, the defect detection method provided by the first aspect of the present invention can be implemented in two parts: an offline training phase and an online analysis phase.

[0084] Please refer to Figure 1 , Figure 1 A flow chart of an online analysis phase of a defect detection method provided according to some embodiments of the present invention is shown.

[0085] like Figure 1 As shown, the online analysis phase of the defect detection method 100 may include step S110: obtaining a target domain sample to be detected;

[0086] The online analysis phase of the defect detection method 100 may further include step S120: inputting the target domain sample to be detected into the neural network detection model to perform defect detection.

[0087] The defect detection system may first obtain target domain samples to be detected, that is, sample data without label information, and then input the obtained target domain samples to be detected into a trained neural network detection model to perform defect detection.

[0088] Please refer to Figure 2 , Figure 2 A flow chart of an offline training phase of a defect detection method provided according to some embodiments of the present invention is shown.

[0089] The defect detection method may also include an offline training phase. Figure 2 As shown, in the offline training stage, training the neural network detection model may include step S210: adding a domain adaptation structure to an initialized basic target detection network to determine a domain adaptation detection network, where the domain adaptation structure includes a feature decoupling domain adaptation module and a bounding box domain adaptation module.

[0090] Preferably, the defect detection system may first input the source domain data set into the basic target detection network for training to determine the initialization of the basic target detection network.

[0091] Please refer to Figure 3 , Figure 3 A schematic diagram of a basic target detection network provided according to a preferred embodiment of the present invention is shown.

[0092] exist Figure 3 In the illustrated embodiment, the basic target detection network 300 may be Faster R-CNN (Faster Region-based Convolutional Network), and the basic network framework may include four parts: a feature extraction network module 310, a region candidate network module 320, an interest domain alignment module 330, and a classification regression module 340. The feature extraction network module 310, the region candidate network module 320, and the interest domain alignment module 330 may be used to form a detector structure of the basic target detection network 300.

[0093] Specifically, the input of the feature extraction network module 310 is a picture, and the output is the features of the picture, namely, FeatureMap. The feature extraction network module 310 can reduce the data dimension without losing important features of the data during the entire target detection process, and extract and organize effective features for use by the subsequent region candidate network module 320 and the fully connected layer.

[0094] The input of the region candidate network module 320 (RPN) is the FeatureMap output by the feature extraction network module 310, and the output is multiple candidate boxes (Proposal) and corresponding target scores. Each candidate box output can be specifically represented by four coordinate values, and the four coordinate values ​​can be the predicted position. The target score can be obtained by binary classification of each candidate box area through the softmax function, and the target score can be used to represent the probability of containing a target in the candidate box (for example, judging whether the candidate box contains a target or not).

[0095] The region of interest alignment module 330 (ROI Alignment) can take the candidate box output by the region candidate network module 320 and the Feature Map output by the feature extraction network module 310 as input, integrate the two to obtain a fixed-size region feature map (Proposal Feature Map), and output it to the subsequent fully connected network for classification.

[0096] The input of the classification and regression module 340 is the regional feature map obtained by the previous layer of interest domain alignment module 330, and the output is the category to which the object in the candidate frame belongs and the precise position of the object in the image. The classification and regression module 340 can classify the image through the softmax function and correct the precise position of the object through the bounding box regression.

[0097] The source domain data set with label information is input into the basic target detection network 300 for training, thereby the defect detection system can obtain a pre-trained parameter model for initializing the basic target detection network, thereby determining the initialization of the basic target detection network.

[0098] Here, the defect detection system can first obtain a public data set, and divide the public data set into a source domain data set and a target domain data set. In a preferred embodiment, the defect detection system can obtain a data set made of flat steel images taken by a high-frequency camera, and the data set made of the flat steel images collects four typical flat steel surface defects, namely pitting (Ps), oil spots (Os), inclusions (In) and scratches (Sc). The defect detection system can also obtain a surface defect data set, and the surface defect data set collects six typical hot-rolled strip surface defects, namely, scale indentation (Rs), plaque (Pa), cracking (Cr), pitting (Ps), inclusions (In) and scratches (Sc). Then, the defect detection system can preprocess the data set made of the flat steel images obtained above and the surface defect data set, extract the three defect categories of pitting (Ps), inclusions (In) and scratches (Sc) contained in the two data sets, and make them into a source domain data set and a target domain data set of the same category but different domains.

[0099] The defect detection method provided by the present invention can realize the cross-domain migration of part defect detection from the source domain dataset to the target domain dataset, for example, the migration from the dataset made of flat steel images to the surface defect dataset; in other non-limiting embodiments, the defect detection method provided by the present invention can also be applied to the cross-domain migration between different types of precision parts in the aerospace field, such as the migration of impeller parts to shell parts. In the cross-domain migration, the defect detection method can also perform high-adaptive migration in the detection of defects such as scratches, marks, built-up edges, and chatter marks.

[0100] Here, the source domain images and the target domain images included in the source domain dataset and the target domain dataset may be referred to as instances or instance data.

[0101] Afterwards, if Figure 2As shown, the defect detection system adds the domain adaptation structure to the initialized basic target detection network to determine the domain adaptation detection network. The domain adaptation structure may include a feature decoupling domain adaptation module and a bounding box domain adaptation module. The feature decoupling domain adaptation module can decompose the inter-domain invariant features and the inter-domain instance invariant features, thereby improving the classification performance of the neural network detection model; the bounding box domain adaptation module can cooperate with the classification and regression module in the initialized basic target detection network to improve the positioning performance of the neural network detection model.

[0102] Preferably, the feature decoupling domain adaptation module may include a shallow feature decoupling domain adaptation module and a deep feature decoupling domain adaptation module; the domain adaptation structure may also include a cascaded RPN domain adaptation module and a classification domain adaptation module.

[0103] Please refer to Figure 4 , Figure 4 A schematic diagram of a domain adaptation detection network provided according to a preferred embodiment of the present invention is shown.

[0104] like Figure 4 As shown, the initialization basic target detection network is based on the basic target detection network 300, including a detector structure and a classification regression module. The detector structure may include a feature extraction network module, a region candidate network module, and an interest domain alignment module. The shallow feature decoupling domain adaptation module 411 and the deep feature decoupling domain adaptation module 412 are configured in the feature extraction network module, the cascaded RPN domain adaptation module 420 is configured in the region candidate network module, and the classification domain adaptation module 430 and the bounding box domain adaptation module 440 are configured in the classification regression module, thereby forming a domain adaptation detection network 400.

[0105] Please refer to Figure 5 , Figure 5 A schematic diagram of a feature decoupling domain adaptation module provided according to a preferred embodiment of the present invention is shown.

[0106] The feature decoupling domain adaptation module can be a shallow feature decoupling domain adaptation module or a deep feature decoupling domain adaptation module. Figure 5 As shown, the feature decoupling domain adaptation module may include a first feature extractor E for extracting inter-domain instance-specific distribution invariant features. di , the second feature extractor E for extracting inter-domain invariant features ds And mutual information loss function. The input of the feature decoupling domain adaptation module is the feature F extracted by the feature extraction network module. b The first feature extractor E di It can be used to extract instance-specific distribution invariance features F between domains di , the second feature extractor E ds A basic domain adaptation module 510 can be cascaded to extract the domain-invariant feature F dsThe basic domain adaptation module 510 can be composed of a GRL (Gradient Reversal Layer) and a domain classification discriminator D ds In addition, the feature decoupling domain adaptation module also includes the mutual information loss function L MI , mutual information loss function L MI It can be used to increase the instance-specific distribution invariance feature F between domains di and domain-invariant features F ds The differences between them are used to achieve feature decoupling capabilities and align instance-level invariant features from different domains to ensure feature consistency between domains and improve the cross-domain performance of the neural network detection model.

[0107] like Figure 4 As shown, when the feature decoupling domain adaptation module is a shallow feature decoupling domain adaptation module 411 and a deep feature decoupling domain adaptation module 412, in order to splice the shallow feature decoupling domain adaptation module 411 and the deep feature decoupling domain adaptation module 412, the feature extraction network module of the initialization basic object detection network can be divided into a shallow feature extraction part And deep feature extraction part Preferably, the feature extraction network module for initializing the basic object detection network can adopt ResNet50.

[0108] In the shallow feature extraction part After that, a shallow feature decoupling domain adaptation module 411 may be externally connected. The input of the shallow feature decoupling domain adaptation module 411 is the shallow feature extraction part. Extracted shallow features The output is the first instance specific distribution invariant feature and the first domain-invariant feature In the deep feature extraction part After that, the deep feature decoupling domain adaptation module 412 can be connected externally to extract the deep feature Extracted deep features As input, the output is instance-specific distribution-invariant features and domain-invariant features Preferably, the feature extraction network module of the domain adaptation detection network 400 further includes a basic domain adaptation module 413 and a basic domain adaptation module 414 cascaded with the feature extraction network module. The basic domain adaptation module 413 and the basic domain adaptation module 414 can be composed of a GRL and a domain classification discriminator D b constituted.

[0109] By splicing the shallow feature decoupling domain adaptation module and the deep feature decoupling domain adaptation module after the middle layer and the last layer of the feature extraction network respectively, the defect detection system can extract the specific distribution of inter-domain instances, thereby enhancing the subsequent classification ability of the network.

[0110] In addition, preferably, the defect detection system can directly replace the region candidate network module of the initialized basic detection network with the cascaded RPN domain adaptation module or add the cascaded RPN domain adaptation module to the region candidate network module of the initialized basic detection network.

[0111] Please refer to Figure 6 , Figure 6 A schematic diagram of a cascaded RPN domain adaptation module provided according to a preferred embodiment of the present invention is shown.

[0112] like Figure 6 As shown in the figure, the input of the cascaded RPN domain adaptation module can be the inter-domain instance-specific distribution invariance feature F output by the feature decoupling domain adaptation module di The cascaded RPN domain adaptation module may include at least two cascaded RPNs. Figure 6 In the illustrated embodiment, the RPN may include a feature extractor 610 and at least one convolutional layer 620. The output of the convolutional layer 620 may be a candidate box 621 or a target score 622. Preferably, the RPN may use adaptive convolution as a feature extraction function to better adapt to the candidate boxes (Proposals) and target features at different stages, thereby improving the accuracy and robustness of the candidate boxes. Thus, the output generated by the cascaded RPN domain adaptation module processed by the cascaded RPN is processed by the non-maximum suppression method (NMS) to finally output a high-accuracy candidate box 640.

[0113] The feature extractor 610 of the RPN may be spliced ​​with a basic domain adaptation module 630, where the basic domain adaptation module 630 may be composed of a GRL and a domain classification discriminator D. By splicing the basic domain adaptation module 630 after the feature extractor 610 of the RPN, the source domain image and the target domain image may be aligned when extracting candidate boxes.

[0114] Please continue to refer to Figure 4 ,exist Figure 4 In the domain adaptation detection network 400 shown, in order to splice the cascaded RPN domain adaptation module 420, the defect detection system can directly replace the region candidate network module of the initialized basic detection network with the cascaded RPN domain adaptation module 420. The input of the cascaded RPN domain adaptation module 420 is the instance-specific distribution invariant feature output by the deep feature decoupling domain adaptation module 412 The output is a high-accuracy candidate box.

[0115] The defect detection system uses a cascaded RPN domain adaptation module to replace the RPN (region candidate network module) in the original structure of the initialized basic detection network, which can enhance the accuracy of candidate box selection while focusing on the invariance features of inter-domain instances.

[0116] In addition, preferably, Figure 4 In the illustrated embodiment, a classification domain adaptation module 430 and a bounding box domain adaptation module 440 are also included in the initialized basic target detection network. The classification domain adaptation module 430 and the bounding box domain adaptation module 440 can be in a cascade relationship in structure. The classification domain adaptation module 430 and the bounding box domain adaptation module 440 have independent model parameters, independent input data distribution, and independent training losses. Therefore, during training, the classification domain adaptation module 430 and the bounding box domain adaptation module 440 can be decoupled for training and do not affect each other during back propagation training.

[0117] The decoupled classification domain adaptation module 430 and the bounding box domain adaptation module 440 can be spliced ​​after the detector structure. There is only data transmission between the classification domain adaptation module 430 and the bounding box domain adaptation module 440, and there is no gradient effect. The fixed-size feature representation extracted by the detector structure As input, the output category label and bounding box label are returned to the detector structure for training of the domain adaptation detection network 400.

[0118] Please refer to Figure 7 and Figure 8 , Figure 7 FIG. 4 shows a schematic diagram of a classification domain adaptation module provided according to a preferred embodiment of the present invention. Figure 8 A schematic diagram of a bounding box domain adaptation module provided according to a preferred embodiment of the present invention is shown.

[0119] like Figure 7 As shown in the figure, the classification domain adaptation module consists of a feature extractor E cls , a class discriminator G cls and a basic domain adaptation module 710. The classification domain adaptation module is implemented by the feature extractor E cls Extract the feature f of each instance and then pass it through the category discriminator G cls Determine the class prediction g for each instance cls .

[0120] like Figure 8 As shown, the bounding box domain adaptation module consists of a feature extractor E reg , a main regressor G reg and an adversarial regressor The bounding box domain adaptation module can be constructed through the feature extractor Ereg Extract the feature f of each instance and then pass it through the main regression G reg Determine the bounding box regression prediction g reg , and through GRL and adversarial regressor Determining bounding box regression predictions Bounding Box Regression Prediction With the main regressor G reg The resulting bounding box regression prediction g reg A confrontation training strategy can be formed: confrontation regression By maximizing and main regressor G reg distance on the target domain while minimizing the distance on the source domain; feature extractor E reg The extracted feature f tries to minimize the distance on the target domain. This adversarial training learning idea can extract the domain invariance of bounding box regression between the target domain and the source domain.

[0121] Therefore, the defect detection system can adaptively add the designed domain adaptation structure to the initialized basic object detection network, thereby determining the domain adaptation detection network.

[0122] Please continue to refer to Figure 2 In the offline training stage, training the neural network detection model may include step S220: inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall loss function of the network reaches a preset target, thereby obtaining a trained neural network model.

[0123] Specifically, Figure 4 As shown, the defect detection system inputs a set of source domain images and target domain images into the domain adaptation detection network for training, and ends the training of the domain adaptation detection network when the overall network loss function reaches a preset target. Here, the overall network loss function may include a first network loss function, a second network loss function, and a third network loss function.

[0124] The first network loss function may include a feature decoupling domain adaptation module loss function, a feature base domain discrimination loss function, and an association consistency loss function. When the feature extraction network is divided into a shallow area and a deep area, the feature decoupling domain adaptation module loss function may include a shallow feature decoupling domain adaptation module loss function and a deep feature decoupling domain adaptation module loss function, and the feature base domain discrimination loss function may include a shallow feature base domain discrimination loss function and a deep feature base domain discrimination loss function. Preferably, the first network loss function may also include a mutual information loss function. The second network loss function may include a cascaded RPN domain adaptation module loss function. The third network loss function may include a classification domain adaptation module loss function and a bounding box domain adaptation module loss function, and preferably, the third network loss function may also include a classification regression loss function.

[0125] Specifically, please refer to Fig. 9 , Fig. 9 A flowchart of training a domain adaptation detection network according to some embodiments of the present invention is shown.

[0126] like Fig. 9 As shown, when training a domain adaptation detection network, the defect detection system may include step S1: inputting a set of source domain images and target domain images into a feature extraction network of the domain adaptation detection network to determine instance-specific distribution invariance features, and calculating a feature decoupling domain adaptation module loss function and a feature basis domain discrimination loss function.

[0127] Please combine Figure 4 , Figure 5 and Fig. 9 The defect detection system may first perform step S11: input a set of source domain images and target domain images into the shallow feature extraction part of the feature extraction network of the domain adaptation detection network 400. To extract shallow features

[0128] Afterwards, the defect detection system can execute step S12: convert shallow features Input to the shallow feature decoupling domain adaptation module 411 to extract the first instance specific distribution invariance feature and the first domain distribution invariance feature The shallow feature decoupling domain adaptation module loss function is determined according to the shallow feature decoupling domain adaptation module 411. The GRL and domain classification discriminator input to the basic domain discrimination module 413 related to the shallow area in the feature extraction module In order to enhance the ability to extract inter-domain invariant features and calculate the shallow feature base domain discriminant loss function.

[0129] like Figure 4 As shown, the defect detection system then performs step S13: shallow feature With the first instance specific distribution invariance feature Add to determine the first feature

[0130] Afterwards, the defect detection system may execute step S14: convert the first feature Input to the deep feature extraction part of the feature extraction network To extract deep features Then execute step S15: transform the deep features Input to the deep feature decoupling domain adaptation module 412 to extract instance-specific distribution invariant features and inter-domain distribution invariance features The deep feature decoupling domain adaptation module loss function can be determined according to the deep feature decoupling domain adaptation module 412. The GRL and domain classification discriminator input to the basic domain discrimination module 414 associated with the deep region in the feature extraction module In order to enhance the ability to extract inter-domain invariant features and calculate the deep feature base domain discriminant loss function.

[0131] Please continue to refer to Figure 4 and Fig. 9 , the defect detection system may include step S2 when training the domain adaptation detection network: transforming instance-specific distribution invariant features The input is sent to the cascaded RPN domain adaptation module 420 to determine high-precision candidate boxes, while keeping the focus on the inter-domain distribution invariance features and calculating the second network loss function, namely the cascaded RPN domain adaptation module loss function.

[0132] Afterwards, the defect detection system can perform step S3 when training the domain adaptation detection network: extract instance-specific distribution invariant features based on the candidate boxes The feature representation And calculate the associated consistency loss function in the first network loss function.

[0133] like Figure 4 As shown, the defect detection system can detect the deep features determined in step S1. Instance-specific distribution invariance features and inter-domain distribution invariance features The high-precision candidate box determined in step S2 is input into the ROIAlignment (RA) layer (i.e., the region of interest alignment module of the initialization basic target detection network), and a fixed-size region corresponding to the deep feature is extracted from the feature map. The first feature representation Fixed-size features corresponding to instance-specific distribution invariance The feature representation and fixed-size features corresponding to inter-domain distribution invariance The second feature representation

[0134] Preferably, when mapping the input feature map and candidate box to the regional feature map, the defect detection system can sample on the floating point coordinates of the regional feature map and accurately calculate the eigenvalues ​​of each sub-grid through bilinear interpolation to eliminate quantization errors, thereby more accurately retaining the feature information.

[0135] Defect detection systems can compute deep features from feature representations and instance-specific distribution invariance features By associating the consistency loss function, the integrity of the instance invariance information can be further maintained.

[0136] Preferably, Figure 4 In the embodiment shown, the training domain adaptation detection network may further include a feature reconstruction process, and accordingly, the overall network loss function may further include a reconstruction loss function. Specifically, the feature reconstruction process may be based on the feature representation. and the second feature representation Perform feature reconstruction and represent it based on the first feature Calculate the reconstruction loss function. According to the feature reconstruction process, the defect detection system can avoid the loss of important feature information during the feature decoupling process.

[0137] Then, the defect detection system can perform step S4 when training the domain adaptation detection network: transform the feature representation Input to the classification domain adaptation module 430 for domain adaptation adversarial training to determine the feature representation containing foreground information And calculate the classification domain adaptation module loss function. By performing domain adaptation adversarial training in the classification domain adaptation module 430, the defect detection system can avoid the feature representation of most target domains. The problem of aligning with the dominant category in the source domain and thus losing category detection accuracy.

[0138] Then, step S5 is performed: the feature representation containing the foreground information is represented as The input is sent to the bounding box domain adaptation module 440 to perform bounding box domain adaptation adversarial training and calculate the bounding box domain adaptation module loss function. Through bounding box domain adaptation adversarial training, the defect detection system can reduce domain differences.

[0139] Then, the defect detection system can repeat steps S1 to S5 until the overall network loss function reaches a preset target, and determine the final trained neural network model.

[0140] Preferably, the first network loss function calculated in step S1 and step S3 may be as shown in formula (1) to formula (5):

[0141]

[0142] L FL = -α(1-p) γ log(p) (2)

[0143]

[0144] Among them, formula (1) can be the shallow feature decoupling domain adaptation module loss function, shallow feature base domain discrimination loss function, deep feature decoupling domain adaptation module loss function, deep feature base domain discrimination loss function corresponding to the shallow feature decoupling domain adaptation module, shallow feature base domain discriminator module, deep feature decoupling domain adaptation module and deep feature base domain discriminator module; k∈[1,2], when k=1, it represents the shallow area, and when k=2, it represents the deep area. There are four domain classification discriminators in total, including domain classification discriminator and domain classification discriminator is a domain classification discriminator of the basic domain adaptation module of the feature extraction network, when k=1, it represents the domain classification discriminator of the basic domain adaptation module 413 related to the shallow region, and when k=2, it represents the domain classification discriminator of the basic domain adaptation module 414 related to the deep region; is the domain classification discriminator of the feature decoupling domain adaptation module. When k=1, it represents the domain classification discriminator of the base domain adaptation module of the shallow feature decoupling domain adaptation module. When k=2, it represents the domain classification discriminator of the base domain adaptation module of the deep feature decoupling domain adaptation module.

[0145] The classification process of the domain classification discriminator can be a binary classification process, and the defect detection system can use the Focal Loss function L FL To constrain the domain classification discriminator so that the domain classification discriminator can correctly distinguish whether the feature comes from the source domain or the target domain. Focal Loss function L FL As shown in formula (2), p is the predicted probability that the domain classification discriminator identifies the sample as the target domain, and α and γ are two adjustable hyperparameters.

[0146] Preferably, if Figure 5 As shown, the feature decoupling domain adaptation module also includes a mutual information loss function L MI , the mutual information loss is calculated in both the deep feature decoupling domain adaptation module and the shallow feature decoupling domain adaptation module. The mutual information loss function L MI As shown in formula (3), n is the number of samples, j is used for counting, j∈[1,n], representing from the first sample to the nth sample. Mutual information loss function L MI By minimizing the mutual information loss function L MI To increase the difference between the invariant features between domains and the instance-specific distribution invariant features between domains, so as to decouple the features. di is the instance-specific distribution invariance feature between domains, F ds is the domain-invariant feature, F′ di is the instance-specific distribution invariance feature between domains after shuffling; here, the instance-specific distribution invariance feature between domains F diand domain-invariant features F ds The features of the same dimension correspond to the same instance or the same dimension, and the instance-specific distribution invariance features F′ between domains after the order is disrupted. di and domain-invariant features F ds T(θ) is a fully connected neural network with θ as parameter, which is responsible for learning the paired features (F di ,F ds ) as the joint distribution, and then sample from the unpaired features (F′ di ,F ds ) is sampled as the marginal distribution to calculate the mutual information loss.

[0147] In addition, the association consistency loss function L RC As shown in formula (4). Specific distribution invariance features for instances across domains A set of fixed-size feature representations extracted by the RA layer, For deep features The first feature representation of a set of fixed size extracted by the RA layer. Through the association consistency loss function L RC ,The defect detection system can further maintain the integrity of instance invariant information.

[0148] Thus, the defect detection system can determine the total loss function of the feature extraction network part, that is, the first network loss function as shown in formula (5). The first network loss function may include L DC , namely, the shallow feature decoupling domain adaptation module loss function, the shallow feature base domain discrimination loss function, the deep feature decoupling domain adaptation module loss function, and the deep feature base domain discrimination loss function; in addition, the total loss function of the feature extraction network part can also include the mutual information loss function L MI And the associated consistency loss function L RC .

[0149] In addition, the second network loss function calculated in step S2, that is, the cascaded RPN domain adaptation module loss function can be shown as formula (6):

[0150]

[0151] Wherein, τ is the number of cascaded RPNs. In a preferred embodiment, the number of cascaded RPNs is 2; λ is the weight coefficient; is α τ The regression loss function of the weight at the τ stage, L cls is the classification loss function, L DC_RPNis the domain adaptation loss function. The classification loss function and the regression loss function have different true values ​​when calculating the loss; when calculating the loss of the source domain image, the real label information is used as the true value, and when calculating the loss of the target domain image, the label information output by the subsequent decoupled classification domain adaptation module and bounding box domain adaptation module is used as the pseudo label true value to calculate the loss of the classification loss function and the regression loss function.

[0152] In addition, in the preferred feature reconstruction process, the reconstruction loss function can be shown as formula (7):

[0153]

[0154] in, are two features used to decouple the features (i.e., the instance-specific distribution invariance features F between domains) di and domain-invariant features F ds ) is restored to the reconstruction module before decoupling, N represents the number of candidate boxes determined in step S2, Respectively represent the deep features from 1 to N The first feature representation Instances of, from 1 to N correspond to the inter-domain distribution invariance features The second feature representation The instances of and the number from 1 to N correspond to instance-specific distribution invariant features The feature representation Instance of .

[0155] The third network loss function may include a classification domain adaptation module loss function and a bounding box domain adaptation module loss function, and preferably, may also include a classification regression loss function.

[0156] Specifically, the classification domain adaptation module loss function in the third network loss function calculated in step S4 can be shown as formula (8) to formula (11):

[0157]

[0158] Among them, in formula (8) is the domain adversarial loss function, f is the feature extracted for each instance, E cls is a feature extractor, i is used for counting, represents the i-th instance feature of the source domain, represents the i-th instance feature of the target domain, N s is the number of source domain samples, N t is the number of samples in the target domain; g cls is the category prediction for each instance, g cls =G cls(f), G cls is a classifier, i.e., a category discriminator; c is the score of each instance predicted as foreground or background by the cascaded RPN domain adaptation module, and w(c) sets the weight according to the score. When the instance is the foreground, w(c) sets a higher weight to reduce the participation of proposals that are neither foreground nor background, and improve the discreteness of the input space in a probabilistic sense; D is a domain classification discriminator, which combines the extracted features and the predicted categories for domain identification and aligns the features in a conditional manner, thereby avoiding alignment of most target domain instances with the dominant categories on the source domain instances.

[0159] In formula (9), is the classification loss function for source domain instances with true label information, L CE is the cross entropy loss function, is the true category label of the source domain instance. Since the target domain instance has no true label information, the classification loss calculation of the target domain instance is not performed. The predicted category label is generated for each target domain instance through formula (11): As pseudo-category labels, the detector structure is trained, where t represents the target domain and N t is the number of samples in the target domain.

[0160] Therefore, the defect detection system can determine the total classification domain adaptation module loss function as shown in formula (10), including the domain adversarial loss function and the classification loss function for source domain instances with true label information Where μ is the weight coefficient.

[0161] The classification domain adaptation module finally inputs the feature representation data of the predicted instance containing foreground information into the bounding box domain adaptation module. The bounding box domain adaptation module loss function in the third network loss function calculated in step S5 can be shown as formula (12) to formula (16):

[0162]

[0163]

[0164] In formula (12), is the bounding box regression loss function for the source domain instance, c is the regression prediction, u is the true value label of the category, v is the true value bounding box offset, x, y represent the center point coordinates of the bounding box, e, h represent the width and height of the bounding box respectively; in formula (14), is the adversarial loss function of the bounding box domain adaptation module, To counter the regressor, G reg is the main regressor, E reg is the feature extractor, is the number of foreground instances in the target domain, is the number of foreground instances in the source domain, f t is the extracted instance feature of the target domain, f s is the extracted source domain instance feature, is the pseudo category label of the target domain instance generated by formula (11) in the classification domain adaptation module, is the true category label of the source domain instance; the adversarial loss function of the bounding box domain adaptation module enables the adversarial regressor With the main regressor G reg The difference in the target domain is maximized, while the difference in the source domain is minimized to measure the difference between domains; in formula (14), The function is shown in formula (13).

[0165] Thus, the defect detection system can determine the loss function of the overall bounding box domain adaptation module. In formula (15), is the total loss function of the bounding box domain adaptation module, which may include the bounding box regression loss function for the source domain instance. And the adversarial loss function of the bounding box domain adaptation module Where η is the weight coefficient.

[0166] In addition, through formula (16), the defect detection system can generate a predicted bounding box label for each target domain instance It can be used as pseudo bounding box labels to input into the detector structure for training.

[0167] The detector structure also includes classification and regression training of target domain instances. Specifically, the defect detection system can use the category pseudo labels and bounding box pseudo labels output by the classification domain adaptation module and the bounding box domain adaptation module as true values ​​to calculate the classification regression loss for training. Classification regression loss function for classification and regression training of target domain instances It can be shown as formula (17):

[0168]

[0169] Among them, L cls and L reg Represent the classification loss function and bounding box loss function respectively, f t is the extracted target domain instance feature, is the pseudo-category label generated for each target domain, is the pseudo bounding box label generated after domain adaptation for each target domain sample, Bounding box labels for bounding box detection without domain adaptation.

[0170] Please continue to refer to Figure 2In the process of executing step S220, the defect detection system can repeatedly calculate each loss function until the overall loss function of the network reaches a preset target, and then stop training the domain adaptation detection network to obtain a trained neural network model.

[0171] Then, the defect detection system may perform step S230: removing the domain adaptation structure in the neural network model to obtain a neural network detection model. Here, the removed domain adaptation structure may be selectively removed according to actual conditions. The obtained neural network detection model may be used to detect the target domain sample of the unlabeled information to be detected acquired by the defect detection system.

[0172] In summary, the defect detection method and defect detection system provided by the present invention can improve the defect detection accuracy on the target domain data set without label information through multi-dimensional decoupling and domain adaptation confrontation methods, and have practical value in cross-domain detection of part defects.

[0173] The following is a specific non-limiting preferred embodiment, based on which the defect detection method and defect detection system proposed in the present invention are further described.

[0174] The defect detection system can preprocess the dataset made of the acquired flat steel images and the surface defect dataset, extract the three defect categories of pitting (Ps), inclusions (In) and scratches (Sc) contained in the two datasets, and make them into source domain datasets and target domain datasets of the same category but different domains. The data extracted from the dataset made of the flat steel images is used as the source domain dataset, and the data extracted from the surface defect dataset is used as the target domain dataset. The training datasets in the source domain dataset and the target domain dataset are used to train the neural network detection model.

[0175] Afterwards, the test data set is used to determine the defect detection accuracy of the neural network detection model. Specifically, the defect detection system can determine the defect detection accuracy of the neural network detection model based on the number of defects detected in the test data set and the number of all defects in the test data set, and use the neural network detection model with the highest accuracy as the neural network detection model for the required high-precision target detection.

[0176] Here, precision is a performance evaluation index of the neural network detection model. Specifically, the defect detection system can use the average precision AP (average precision) as the performance evaluation index of the neural network detection model. The average precision AP mainly includes two values: precision and recall. Precision indicates the proportion of samples that actually belong to the category in the samples predicted to be positive in different categories. The precision can be shown as formula (18):

[0177]

[0178] Among them, TP (True Positive) is the number of samples of defective targets correctly identified; FN (False Negative) is the number of samples of defective targets not identified; FP (False Positive) is the number of samples of defective targets incorrectly identified.

[0179] The recall rate represents the ratio of the number of correctly detected targets to the total number of targets in the test set. The recall rate can be expressed as formula (19):

[0180]

[0181] The denominator is TP (True Positive) + FN (False Negative), which is the total number of defective samples.

[0182] Therefore, the average accuracy AP can be determined by precision and recall. The average accuracy AP is shown in formula (20):

[0183]

[0184] in, is the area under the PR (Precision-Recall) curve, M is the number of different recall levels, p(m) is the precision at the mth recall level, and Δr(m) is the difference between the mth recall level and the previous recall level.

[0185] Furthermore, the defect detection system can evaluate the neural network detection model through the mean average precision MAP (mean average precision), which represents the mean of the average accuracy of all categories in the neural network detection model. The calculation formula of the mean average precision MAP can be shown as formula (21):

[0186]

[0187] Among them, a is the a-th category and Z is the total number of categories.

[0188] Please refer to Fig.10 , Fig.10 The results show the accuracy of defect detection on target domain data without label information by the defect detection method provided in the preferred embodiment of the present invention.

[0189] The technicians compared the defect detection method 1000 provided by the present invention with the existing models 1001, 1002, 1003 and 1004. The existing model 1001 is a baseline method model, that is, a model in which the basic target detection network directly predicts the unlabeled target domain after only being trained in the labeled source domain. Models 1002, 1003 and 1004 are the DA-Faster method (Domain Adaptive Faster, a domain adaptation method), the SW-DA method (Strong-Weak Distribution Alignment, a domain adaptation method) and the D-adapt method (Decoupled Adaptation, a domain adaptation method), respectively. Fig.10 As shown, when the category of model 1001 is pitting (Ps), the average accuracy is 29.1; when the category is inclusion (In), the average accuracy is 23.1; when the category is scratch (Sc), the average accuracy is 45.3; and the average precision mean MAP is 32.5. When the category of model 1002 is pitting (Ps), the average accuracy is 32.7; when the category is inclusion (In), the average accuracy is 30.2; when the category is scratch (Sc), the average accuracy is 61.8; and the average precision mean MAP is 40.9. When the category of model 1003 is pitting (Ps), the average accuracy is 34.8; when the category is inclusion (In), the average accuracy is 32; when the category is scratch (Sc), the average accuracy is 63.4; and the average precision mean MAP is 43.4. When the category of model 1004 is pitting (Ps), the average accuracy is 45.2; when the category is inclusion (In), the average accuracy is 34; when the category is scratch (Sc), the average accuracy is 74.4; the average precision mean MAP is 51.2. When the category of the defect detection method 1000 provided by the present invention is pitting (Ps), the average accuracy is 48.9; when the category is inclusion (In), the average accuracy is 36.1; when the category is scratch (Sc), the average accuracy is 78.5; the average precision mean MAP is 54.5. It can be seen that the defect detection method provided by the present invention has obvious improvements in the detection of each defect category, and realizes the improvement of the accuracy of defect detection on the target domain data set without label information, and has practical value in cross-domain detection of part defects.

[0190] Please refer to Fig.11 , Fig.11 A schematic diagram of a defect detection system provided according to some embodiments of the present invention is shown.

[0191] The defect detection system provided by the present invention can be used as follows Fig.11As shown, the defect detection system 1100 may be configured with a memory 1110 and a processor 1120. The memory 1110 includes but is not limited to the computer-readable storage medium 1111 provided in the third aspect of the present invention, on which computer instructions are stored. The processor 1120 is connected to the memory 1110 and is configured to execute the computer instructions stored in the memory 1110 to implement the defect detection method provided in the first aspect of the present invention.

[0192] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0193] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0194] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0195] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0196] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0197] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0198] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A defect detection method, characterized in that: Includes steps: Obtain target domain samples to be detected; as well as Inputting the target domain sample to be detected into a neural network detection model to perform defect detection; The step of training the neural network detection model includes: Adding a domain adaptation structure to an initialized basic object detection network to determine a domain adaptation detection network, wherein the domain adaptation structure includes a feature decoupling domain adaptation module and a bounding box domain adaptation module; Inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall loss function of the network reaches a preset target, thereby obtaining a trained neural network model; The domain adaptation structure in the neural network model is removed to obtain the neural network detection model.

2. The defect detection method according to claim 1, characterized in that: The step of training the neural network detection model also includes: The source domain dataset is input into the basic object detection network for training to determine the initialized basic object detection network.

3. The defect detection method according to claim 1, characterized in that: The feature decoupling domain adaptation module includes a shallow feature decoupling domain adaptation module and a deep feature decoupling domain adaptation module, and the domain adaptation structure also includes a cascade RPN domain adaptation module and a classification domain adaptation module.

4. The defect detection method according to claim 3, characterized in that: The initialized basic target detection network includes a detector structure and a classification regression module, wherein the detector structure includes a feature extraction network module, a region candidate network module and an interest domain alignment module. The step of adding the domain adaptation structure to the initialized basic target detection network to determine the domain adaptation detection network includes: The shallow feature decoupling domain adaptation module and the deep feature decoupling domain adaptation module are configured in the feature extraction network module, the cascade RPN domain adaptation module is configured in the region candidate network module, and the classification domain adaptation module and the bounding box domain adaptation module are configured in the classification regression module to determine the domain adaptation detection network.

5. The defect detection method according to claim 4, characterized in that: The feature decoupling domain adaptation module includes a first feature extractor for extracting inter-domain instance-specific distribution invariant features and a second feature extractor for extracting inter-domain invariant features.

6. The defect detection method according to claim 5, characterized in that: The shallow feature decoupling domain adaptation module is configured in the shallow feature extraction part of the feature extraction network, and the deep feature decoupling domain adaptation module is configured in the deep feature extraction part of the feature extraction network.

7. The defect detection method according to claim 4, characterized in that: The cascaded RPN domain adaptation module includes two cascaded RPNs, and the feature extractor of the RPN is spliced ​​with the basic domain adaptation module.

8. The defect detection method according to claim 7, characterized in that: The RPN uses adaptive convolution as the feature extraction function.

9. The defect detection method according to claim 4, characterized in that: The classification domain adaptation module and the bounding box domain adaptation module are in a cascade relationship in structure.

10. The defect detection method according to claim 9, characterized in that: The classification domain adaptation module includes a feature extractor, a category discriminator and a basic domain adaptation module.

11. The defect detection method according to claim 9, characterized in that: The bounding box domain adaptation module includes a feature extractor, a primary regressor and an adversarial regressor.

12. The defect detection method according to claim 4, characterized in that: The overall network loss function includes a first network loss function, a second network loss function and a third network loss function. The first network loss function includes a feature decoupling domain adaptation module loss function, a feature basis domain discrimination loss function and an association consistency loss function. The second network loss function includes a cascaded RPN domain adaptation module loss function. The third network loss function includes a classification domain adaptation module loss function and a bounding box domain adaptation module loss function.

13. The defect detection method according to claim 12, characterized in that: The feature decoupling domain adaptation module loss function includes a shallow feature decoupling domain adaptation module loss function and a deep feature decoupling domain adaptation module loss function. The feature basic domain discrimination loss function includes a shallow feature basic domain discrimination loss function and a deep feature basic domain discrimination loss function. The first network loss function also includes a mutual information loss function.

14. The defect detection method according to claim 12, characterized in that: The third network loss function includes a classification regression loss function.

15. The defect detection method according to claim 12, characterized in that: The step of inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall loss function of the network reaches a preset target to obtain a trained neural network model comprises: S1: Input a set of source domain images and target domain images into the feature extraction network of the domain adaptation detection network to determine instance-specific distribution invariance features, and calculate the feature decoupling domain adaptation module loss function and the feature base domain discrimination loss function; S2: Inputting the instance-specific distribution invariance feature into the cascaded RPN domain adaptation module to determine the candidate box and calculate the second network loss function; S3: extracting a feature representation corresponding to the instance-specific distribution invariance feature according to the candidate box, and calculating the association consistency loss function; S4: inputting the feature representation into the classification domain adaptation module to perform domain adaptation adversarial training to determine the feature representation containing foreground information, and calculating the loss function of the classification domain adaptation module; S5: inputting the feature representation containing the foreground information into the bounding box domain adaptation module, performing bounding box domain adaptation adversarial training, and calculating the loss function of the bounding box domain adaptation module; and S6: Repeat steps S1 to S5 until the overall network loss function reaches a preset target.

16. The defect detection method according to claim 15, characterized in that: The step S1 comprises: S11: inputting the set of source domain images and target domain images into a shallow feature extraction part of the feature extraction network of the domain adaptation detection network to extract shallow features; S12: Inputting the shallow feature into the shallow feature decoupling domain adaptation module to extract the first instance specific distribution invariance feature, and calculating the shallow feature decoupling domain adaptation module loss function and the shallow feature base domain discrimination loss function; S13: adding the shallow feature to the first instance-specific distribution invariant feature to determine a first feature; S14: inputting the first feature into the deep feature extraction part of the feature extraction network to extract deep features; S15: Input the deep features into the deep feature decoupling domain adaptation module to extract instance-specific distribution invariance features, and calculate the deep feature decoupling domain adaptation module loss function and the deep feature base domain discrimination loss function.

17. The defect detection method according to claim 16, characterized in that: The overall network loss function also includes a reconstruction loss function. The step of inputting a set of source domain images and target domain images into the domain adaptation detection network for training until the overall network loss function reaches a preset target and obtaining a trained neural network model also includes a feature reconstruction process. The step S15 comprises: inputting the deep features into the deep feature decoupling domain adaptation module to extract inter-domain invariant features; The feature reconstruction process comprises the steps of: Extracting, according to the candidate box, a first feature representation and a second feature representation corresponding to the deep feature and the inter-domain distribution invariance feature; Feature reconstruction is performed on the feature representation and the second feature representation, and a reconstruction loss function is calculated based on the first feature representation.

18. A defect detection system, characterized in that: include: a memory having computer instructions stored thereon; as well as A processor is connected to the memory and configured to execute computer instructions stored in the memory to implement the defect detection method according to any one of claims 1 to 17.

19. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a processor, the defect detection method according to any one of claims 1 to 17 is implemented.

Images