Industrial defect detection model training method, detection method, robot and storage medium

Abstract

The embodiment of the invention discloses an industrial defect detection model training method, an industrial defect detection method, an industrial robot and a computer readable storage medium. The method comprises the following steps: taking a simulation defect image as the input of a pre-constructed pre-training network model for processing, obtaining an output sample set, and performing a potential positive sample elimination operation to obtain a second negative sample set; performing iterative adjustment on the pre-training network model through a calculation result output by the loss function calculation; marking the pre-trained network model meeting the first training condition as a first model, and transforming to obtain a second model; and training the second model by using the real defect image set to obtain an industrial defect detection model. Therefore, the industrial simulation image is used as the training data for training, and the defect industrial image features are learned more accurately. The problem of low negative sample quality caused by class conflict and random sampling is relieved through potential positive sample rejection, and the classification performance is improved through transfer learning and model fine tuning.

Description

Technical Field

[0001] The present application belongs to the technical field of industrial defect classification, and in particular, relates to an industrial defect detection model training method, an industrial defect detection method, an industrial robot and a computer-readable storage medium. Background Art

[0002] Industrial defect detection has long been one of the most important research areas in the field of industrial vision. In recent years, with the popularity of deep learning in computer vision tasks, industrial defect classification methods based on deep learning have also developed rapidly and gradually become mainstream. Thanks to the powerful feature extraction capabilities and high-dimensional data representation capabilities of convolutional neural networks, deep learning-based methods can automatically learn features that are difficult to design manually, which not only saves the cost of manually designed features, but also greatly improves classification accuracy.

[0003] However, most existing methods of trained classification models do not consider the domain gap between natural images and industrial defect images, and do not effectively utilize the domain knowledge of industrial defect images. At the same time, the class conflict problem in traditional instance-based discriminant class contrast learning may affect the linear separability of features learned by the model, thereby affecting the quality of learned image representation. Therefore, how to improve the image classification for industrial defects is a technical problem that needs to be solved urgently by those skilled in the art.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the invention

[0005] Based on this, it is necessary to propose an industrial defect detection model training method, an industrial defect detection method, an industrial robot and a computer-readable storage medium to address the above problems, which can effectively train the industrial defect detection model for accurate detection.

[0006] The present application solves the technical problem by adopting the following technical solutions: The present application provides an industrial defect detection model training method, comprising the following steps: obtaining a simulated defect image, and using the simulated defect image as an input of a pre-built pre-trained network model for processing; obtaining a sample set output by the pre-trained network model, the sample set including a positive sample pair and a first negative sample set; performing a potential positive sample elimination operation on the first negative sample set to obtain a second negative sample set; using the positive sample pair and the second negative sample set as inputs of a loss function for processing, obtaining a calculation result of the loss function calculation output to iteratively adjust the pre-trained network model until a first training condition is met; marking the pre-trained network model that meets the first training condition as a first model; obtaining a second model based on the first model transformation; obtaining a set of real defect images, and using the set of real defect images as a second model input to train the second model; when the second model meets the second training condition, marking the second model as an industrial defect detection model.

[0007] In an optional embodiment of the present application, obtaining a simulated defect image includes: obtaining an input image, where the input image is a captured image of a real mechanical product; randomly obtaining at least one simulated noise from an industrial defect category library, where each simulated noise corresponds to an industrial defect, and the industrial defects include dimensional deviations, surface defects, assembly problems, and material defects; adding at least one simulated noise to the input image through random data enhancement to obtain a pair of simulated defect sub-images, and summarizing all pairs of simulated defect sub-images to obtain a simulated defect image.

[0008] In an optional embodiment of the present application, a simulated defect image includes a first sub-image of simulated defects and a second sub-image of simulated defects, and the first sub-image of simulated defects and the second sub-image of simulated defects are obtained by processing an image; the pre-trained network model includes a first sub-network and a second sub-network, and the first sub-network and the second sub-network have the same network structure, including a backbone network and a projection head; the simulated defect image is used as input of a pre-built pre-trained network model for processing, including: the first sub-network uses the first sub-image of simulated defects as input for processing, and the second sub-network uses the second sub-image of simulated defects as input for processing; the backbone networks of the first sub-network and the second sub-network encode the input simulated defect image into a potential representation; the projection heads of the first sub-network and the second sub-network project the potential representation into a feature vector, and characterize the sample set in the form of the feature vector.

[0009] In an optional embodiment of the present application, a sample set output by a pre-trained network model is obtained, including: the feature vector output by the first subnetwork is called an anchor sample, the feature vector output by the second subnetwork is called a positive sample, a pair of anchor samples and positive samples are regarded as a group of positive sample pairs, and the anchor samples and positive samples attract each other in the sample space; all positive samples except the positive sample itself are marked as negative samples corresponding to the positive sample, and all negative samples are aggregated to obtain a first negative sample set corresponding to the positive sample; all positive sample pairs and the corresponding first negative sample set are aggregated to obtain a sample set.

[0010] In an optional embodiment of the present application, after performing a potential positive sample elimination operation on the first negative sample set, a second negative sample set is obtained, including: the sample set has a pseudo-label, and the pseudo-label is used to indicate the industrial defect corresponding to the industrial defect simulation image classified and identified by the pre-trained network model; according to the pseudo-label, the negative samples in the first negative sample set are stored in a feature queue; the feature queue includes multiple columns of sub-teams, each column of sub-team corresponds to a pseudo-label, and each column of sub-team stores a preset number of negative samples; for the first negative sample set corresponding to each positive sample, the negative samples in the same sub-team as the positive sample are eliminated from the first negative sample set, and the remaining negative samples are marked as the second negative sample set; the second negative sample and the positive sample pair are mutually exclusive in the sample space.

[0011] In an optional embodiment of the present application, the pre-trained network model includes a first sub-network and a second sub-network, and the positive sample pair includes an anchor sample and a positive sample; the positive sample pair and the second negative sample set are used as inputs of a loss function for processing, and the calculation result of the loss function calculation output is obtained to iteratively adjust the pre-trained network model until the first training condition is met, including: the loss function is expressed as: ; In the above formula, is the anchor sample, is a positive sample, is the second negative sample set, is the preset temperature coefficient; the loss function calculated after each iteration is marked as the loss function value, the current number of iterations is obtained, and the loss function value and the number of iterations are summarized to obtain the calculation result; according to the calculation result, the parameters of the first sub-network are updated by gradient, and the parameters of the second sub-network are updated by momentum. The updating process is: ; In the above formula, are the parameters of the second sub-network, are the parameters of the first sub-network, is a preset hyperparameter; the first training condition includes: the loss function value is less than or equal to a preset acceptable threshold; and / or the number of iterations is greater than or equal to a preset maximum number of iterations.

[0012] In an optional embodiment of the present application, the calculation result of the loss function calculation output is obtained to iteratively adjust the pre-trained network model, including: after each round of iteration, obtaining the pseudo-label of the sample set; based on the pseudo-label, storing the negative samples in the sub-queue corresponding to the feature queue; in the current round of iterative training, selecting the negative samples and simulated defect images in the feature queue obtained in the previous round as the input of the pre-trained network model for processing; and updating the feature queue with the negative samples processed in the current round.

[0013] In an optional embodiment of the present application, a second model is obtained by transforming the first model, including: obtaining a backbone network and training weights of the first model; using the backbone network as an encoder of the second model, and initializing the encoder according to the training weights; constructing a classification head of the second model, the classification head adopts a fully connected layer structure with an input of 512 and an output of 6; and initializing the classification head with a preset mean and variance; a network model obtained by constructing the initialized encoder and classification head is marked as a second model; training the second model, including: inputting a real defect image into the second model for processing, obtaining the processed detection result, and all images in the real defect image set have labels; calculating the accuracy of the second model in the current round according to the detection result and the label; and adjusting the classification head of the second model according to the accuracy training.

[0014] The present application also provides an industrial defect detection method, including: training an industrial defect detection model according to the aforementioned method; obtaining an image to be detected as an input of the industrial defect detection model, and obtaining a classification result output by the industrial defect detection model after processing by the industrial defect detection model.

[0015] The present application also provides an industrial robot, comprising a processor and a memory; the processor is used to execute a computer program stored in the memory to implement the aforementioned industrial defect detection method.

[0016] The present application also provides a computer-readable storage medium storing a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0017] The embodiments of the present application have the following beneficial effects: This application obtains industrial simulation images as training data, and then guides the model to focus on different defect types, so as to more accurately learn the characteristics of industrial images with different defects, which helps to accurately classify. In addition, by eliminating potential positive samples, the problem of low quality of negative samples caused by class conflicts and random sampling is alleviated. And use the pre-training of simulated images, and the transfer learning or feature extraction after pre-training to apply to the classification task of real industrial defect images, and improve the performance of downstream classification tasks through further model fine-tuning to improve the accuracy of classification.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following preferred embodiments are specifically cited and described in detail with the accompanying drawings. It should be understood that the above general description and the detailed description below are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of an industrial defect detection model training method provided in an embodiment.

[0021] Figure 2 A schematic diagram of the architecture of a pre-trained network model provided in one embodiment.

[0022] Figure 3 A schematic diagram of the structure of a projection head provided by an embodiment.

[0023] Figure 4 A schematic flow chart of an industrial defect detection method provided in an embodiment.

[0024] Figure 5 The present invention is a schematic block diagram of the structure of an industrial robot provided by an embodiment. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] Classification algorithms are divided into traditional image processing methods, deep learning methods and some auxiliary technologies according to different classification methods. The traditional image processing methods mainly include statistical methods, spectral methods, model methods and morphological processing. The deep learning methods can be divided into three cases according to the amount of labeled data: when all data are labeled (such as labels, used to characterize the attributes of the corresponding data, such as specific classification types, etc.), the fully supervised learning method is mainly used; when the data is not labeled, the pixel-level and feature-level unsupervised learning methods are mainly used; when only a small amount of data is labeled, small sample learning, weakly supervised learning, semi-supervised learning and self-supervised learning methods can be used; when all data are not labeled, it is divided into pixel-level image reconstruction methods, image restoration methods, or feature-level deep one-class classification or feature distance measurement. On the basis of these classification methods, some auxiliary technologies are also used to improve the performance of the model, such as multimodal fusion technology, data enhancement and model compression.

[0027] For self-supervised learning, the main approach is to use proxy tasks to mine supervisory signals from large-scale unlabeled data, and train the network through the constructed supervisory signals to learn representations that are valuable for downstream tasks. The learned representations can be directly applied to downstream tasks or optimized through network fine-tuning. Even in downstream tasks with limited data, representations trained based on a large amount of unlabeled data can still show good performance. However, the existing deep learning for industrial defect classification has the following problems.

[0028] First, the domain knowledge of industrial defect images is not effectively utilized. Most existing methods do not consider the domain gap between natural images and industrial defect images. The focus area of ​​natural images is generally in the center of the image, while industrial defect images usually focus on features such as the shape, size, and texture of the defect, which may be distributed in any area of ​​the industrial image. Therefore, it is possible to consider guiding the model to focus on these features by applying domain knowledge.

[0029] Second, the class conflict problem in traditional instance-discriminative class contrastive learning may affect the linear separability of features learned by the model. Most existing contrastive learning frameworks use instance discrimination as a proxy task, which regards each instance as a different category. Among them, positive sample pairs are defined as two enhanced views from the same instance, and negative samples are defined as samples from different instances, regardless of their semantic content or visual features. This strategy may cause the model to keep samples that belong to the same category semantically far away in the feature space, thereby affecting the quality of learned image representation.

[0030] To solve the above technical problems, the present application proposes an industrial defect detection model training method, and an industrial defect detection method using the trained industrial defect detection model. Figure 1~Figure 4 , including steps S110~S140, and steps S210~S220.

[0031] Step S110: Acquire a simulated defect image, and use the simulated defect image as an input of a pre-built pre-trained network model for processing.

[0032] In one embodiment, obtaining a simulated defect image includes: obtaining an input image, where the input image is a captured image of a real mechanical product; randomly obtaining at least one simulated noise from an industrial defect category library, where each simulated noise corresponds to an industrial defect, and the industrial defects include dimensional deviation, surface defect, assembly problem, and material defect; adding at least one simulated noise to the input image through random data enhancement to obtain a pair of simulated defect sub-images, and summarizing all pairs of simulated defect sub-images to obtain a simulated defect image.

[0033] In one embodiment, unlike natural image classification, the objects of industrial defect classification are usually specific industrial products, such as mechanical parts, electronic products, plastic products, etc. These products have clear shapes, sizes and structural features. For specific industrial products, their defect types are relatively certain. This application simulates different types of industrial defects by using specific data enhancement operation classes to train the model to identify different types of industrial defects.

[0034] In order to simulate the real situation, at least one simulated noise can be randomly obtained from the industrial defect category library. Each simulated noise corresponds to an industrial defect. The corresponding industrial defect is simulated by adding the corresponding noise to the input image. The input image can be a captured image of a real mechanical product. The industrial defect category library stores preset common defect types, which can include but are not limited to size deviation, surface defects, assembly problems, material defects, etc.

[0035] (1) Dimensional deviation Mechanical products usually have high requirements for dimensional accuracy. Dimensional deviation may lead to poor fit between parts and components, affecting the overall performance and reliability of the machine. For example, if the clearance between the shaft and the bearing is too large or too small, it will cause vibration, increased noise, and even mechanical failure. Dimensional deviation may be caused by machining errors, tool wear, loose fixtures, etc. During the machining process, the machining process and equipment accuracy need to be strictly controlled to ensure that the dimensions meet the design requirements.

[0036] (2) Surface defects The surface quality of mechanical products has an important impact on their performance and life. Common surface defects include scratches, cracks, rust, etc. Scratches may be caused during processing, transportation or assembly, which will affect the appearance and surface finish of the product. Severe scratches may also reduce the strength and wear resistance of the product. Cracks are a serious defect that may cause mechanical products to break during use. The generation of cracks may be related to factors such as material quality, processing technology, and stress concentration. Rust will reduce the surface quality and corrosion resistance of mechanical products and affect their service life. Rust is usually caused by environmental factors (such as moisture, corrosive gases, etc.) or improper protective measures.

[0037] (3) Assembly issues Mechanical products are usually composed of multiple parts, and defects in the assembly process may cause product performance degradation or failure. Common assembly problems include loose parts, misalignment, uneven gaps, etc. Loose parts may be caused by untightened or loose fasteners, uneven mating surfaces, etc. Loose parts will generate vibration and noise during the operation of the machine, and in severe cases may cause parts to fall off and cause safety accidents. Misalignment refers to the position deviation of parts due to improper installation during the assembly process. Misalignment may affect the movement accuracy and performance of the machine, and even cause the machine to fail to work properly.

[0038] (4) Material defects The material quality of mechanical products directly affects their performance and life. Common material defects include inclusions, pores, and looseness. Inclusions refer to non-metallic impurities present in the material, such as oxides, sulfides, etc. Inclusions will reduce the strength and toughness of the material, increase the brittleness of the material, and easily cause the material to break during use. Porosity refers to tiny holes in the material, which will reduce the density and strength of the material and affect the sealing and corrosion resistance of the material. Looseness refers to the uneven organization inside the material, with more pores and defects. Looseness will reduce the strength and toughness of the material, and easily cause the material to deform and break during use.

[0039] For different defects, different noises can be added to the input image to simulate different defects. Size deviations can be simulated by randomly scaling the input image slightly; surface defects can be simulated by adding specific textures or lines to the input image; assembly problems can be simulated by adding image perturbations to the input image; material defects can be simulated by adding spots of different shapes and colors to the input image. In addition, one or more noises can be added during the adding process to simulate more types of defects. Further, the adding process can use random data enhancement to divide the original simulated defect image into two sub-images, including a simulated defect first sub-image and a simulated defect second sub-image. The simulated defect first sub-image and the simulated defect second sub-image are obtained by processing one image, which will facilitate subsequent processing.

[0040] Random data enhancement may include but is not limited to: 1. Geometric transformation: Random flip: horizontal or vertical flip; Random rotation: random rotation of a certain angle (such as −30° to 30°); Random cropping: cropping random areas of the image; Random scaling: randomly adjusting the size of the image. 2. Color transformation: Brightness adjustment: randomly changing the brightness of the image; Contrast adjustment: randomly adjusting the contrast; Hue change: changing the hue of the image; Saturation adjustment: randomly adjusting the saturation of the image. 3. Blur and noise: Random blur: apply Gaussian blur or other blur effects; Random noise: add Gaussian noise, salt and pepper noise, etc. 4. Other enhancements: Cutout: randomly block an area in the image; Mixup: randomly weighted superposition of two images; CutMix: replace part of an image with the corresponding area of ​​another image.

[0041] It is also worth noting that the present application aims to process an image into two similar defect images, namely a first sub-image of simulated defects and a second sub-image of simulated defects, through random data enhancement for subsequent training. At the same time, the method provided by the present application adopts a self-supervised method for training. As mentioned above, self-supervision is a training method based on a small amount of labeled data. Therefore, some images in the obtained simulated defect images are labeled to characterize the industrial defect category simulated by the simulated defect images. Moreover, the number of simulated defect images with labels is much smaller than that of unlabeled simulated defect images. Training can be completed with a small number of labels, which reduces the process of manual labeling and improves the efficiency of training.

[0042] In one embodiment, a simulated defect image includes a first sub-image of simulated defects and a second sub-image of simulated defects, and the first sub-image of simulated defects and the second sub-image of simulated defects are obtained by processing an image; the pre-trained network model includes a first sub-network and a second sub-network, and the first sub-network and the second sub-network have the same network structure, including a backbone network and a projection head; the simulated defect image is used as an input of a pre-constructed pre-trained network model for processing, including: the first sub-network uses the first sub-image of simulated defects as input for processing, and the second sub-network uses the second sub-image of simulated defects as input for processing; the backbone networks of the first sub-network and the second sub-network encode the input simulated defect image into a potential representation; the projection heads of the first sub-network and the second sub-network project the potential representation into a feature vector, and characterize the sample set in the form of the feature vector.

[0043] In one embodiment, the structure of the pre-trained network model can be referred to Figure 2 As shown. Figure 2As shown in FIG. 1 , the pre-trained network model includes a first sub-network and a second sub-network. The first sub-network and the second sub-network have the same network structure, both including a backbone network and a projection head. According to the above description, it can be known that the simulated defect image is obtained by random data enhancement, and two similar images can be obtained from one image. In other words, the simulated defect image actually includes a simulated defect first sub-image and a simulated defect second sub-image, wherein the simulated defect first sub-image is marked as , used as the input of the first sub-network; the second sub-graph of the simulated defect is marked as , used as the input of the second sub-network.

[0044] Encoder in the first sub-network backbone network The first sub-graph of the simulated defect Encoded as the first latent representation, labeled ; Encoder in the second sub-network backbone network The second sub-graph of the simulated defect Encoded as the second latent representation, labeled Among them, the encoder in the backbone network in the first sub-network , and the encoder in the second subnetwork Specifically, the network architecture of ResNet18 (Residual Network, ResNet, with a depth of 18 layers) can be used.

[0045] Furthermore, the first sub-network projection head The first potential representation Project to generate projections ( Indicates that the sample has been through the industrial defect category library data augmentation); the second sub-network projection head The first potential representation Project to generate projections Among them, for the projection head and projection head With the same architecture, you can refer to the architecture of the projection head Figure 3 Projection and projection In fact, they are all expressed in the form of feature vectors, which are used to characterize the sample set.

[0046] Step S120: obtaining a sample set output by the pre-trained network model, the sample set including positive sample pairs and a first negative sample set; performing a potential positive sample elimination operation on the first negative sample set to obtain a second negative sample set.

[0047] In one embodiment, a sample set output by a pre-trained network model is obtained, including: the feature vector output by the first subnetwork is called an anchor sample, the feature vector output by the second subnetwork is called a positive sample, a pair of anchor samples and positive samples are regarded as a group of positive sample pairs, and the anchor samples and positive samples attract each other in the sample space; all positive samples except the positive sample itself are marked as negative samples corresponding to the positive sample, all negative samples are aggregated to obtain a first negative sample set corresponding to the positive sample; all positive sample pairs and the corresponding first negative sample set are aggregated to obtain a sample set.

[0048] In one embodiment, it can be seen from the above description that the output of the pre-trained network model is actually a feature vector, which can be used to represent the sample set in the form of a feature vector, wherein the feature vector output by the first sub-network is It is called the anchor sample; the feature vector output by the second sub-network It is called a positive sample. A pair of anchor samples and positive samples is regarded as a set of positive sample pairs. The anchor samples and positive samples attract each other in the sample space. For the selection of negative samples, for a positive sample, all positive samples except itself are regarded as negative samples. By summarizing all negative samples, we can get the positive sample pair and the corresponding first negative sample set.

[0049] In one embodiment, after performing a potential positive sample elimination operation on the first negative sample set, a second negative sample set is obtained, including: the sample set has a pseudo-label, and the pseudo-label is used to indicate the industrial defect corresponding to the industrial defect simulation image classified and identified by the pre-trained network model; according to the pseudo-label, the negative samples in the first negative sample set are stored in a feature queue; the feature queue includes multiple columns of sub-teams, each column of sub-team corresponds to a pseudo-label, and each column of sub-team stores a preset number of negative samples; for the first negative sample set corresponding to each positive sample, the negative samples in the same sub-team as the positive sample are eliminated from the first negative sample set, and the remaining negative samples are marked as the second negative sample set; the second negative sample and the positive sample pair are mutually exclusive in the sample space.

[0050] In one embodiment, instance discriminant class contrastive learning is used as a baseline model, in which the network is trained to recognize two enhanced views of the same instance (anchor and positive sample) while distinguishing other instances (negative samples). Although instance discriminant class contrastive learning has good linear separability, which is friendly to classification tasks, this method regards each instance as a different category, which inevitably leads to category conflict problems (which means that even for very similar instances, they will still be regarded as negative samples), thereby affecting the quality of the learned image representation. Among all negative samples, the most difficult 0.1% of negative samples are not only unnecessary, but also harmful. The definition of the so-called difficult negative samples here is a negative sample that is very similar to the anchor in semantics. These negative samples may be potential positive samples. Therefore, in the process of comparison, such negative samples can be eliminated from the first negative sample set in this application through a potential positive sample elimination operation.

[0051] The specific operation is to assign corresponding pseudo labels to the sample set when the sample set is obtained. Pseudo labels are a type of labels used to characterize the industrial defects corresponding to the industrial defect simulation images classified and identified by the pre-trained network model. According to the pseudo labels, the corresponding negative samples are stored in the feature queue. The feature queue is a pre-built repository for storing feature vectors in the form of a queue data structure, and it can be seen that the feature vector is a representation form of the sample. There are a total of Team Liezi, is a preset real number. Each column of sub-teams corresponds to a pseudo-label, and each column of sub-teams stores a preset number of negative samples. And the negative samples in each column of sub-teams will be kept updated in the subsequent iterative training process to ensure that the features in the queue are the latest feature vectors. The original first negative sample set is all positive samples except itself. Potential positive sample elimination, that is, to eliminate the negative samples in the first negative sample set that are in the same sub-team as the positive samples, and mark the remaining negative samples as the second negative sample set. In other words, assuming that Positive samples , respectively , … For the Positive samples , the first negative sample set is ~ Medium All samples except for m-1 elements. By setting the feature queue, all samples , … Then, according to their corresponding pseudo labels, they are stored in the queues respectively. ~ In. Assume The sub-team is , then they will belong to The samples of are removed from the first negative sample set; or, the second negative sample set is, except All samples in other subteams except . This makes the positive samples and the negative samples in the second negative sample set mutually exclusive in the feature space, and removes the potential positive samples that may exist therein, thereby reducing the impact of category conflicts on model performance.

[0052] Step S130: using the positive sample pair and the second negative sample set as inputs of the loss function for processing, obtaining the calculation result of the loss function calculation output to iteratively adjust the pre-trained network model until the first training condition is met; marking the pre-trained network model that meets the first training condition as the first model.

[0053] In one embodiment, in the iterative process, in order to achieve the goal of self-supervised representation learning, the present application may use the standard InfoNCE loss to maximize the similarity between positive sample pairs and minimize the similarity between negative sample pairs, that is, the positive sample pair and the second negative sample set are used as inputs of the loss function for processing, where the loss function is expressed as: (1) In formula (1), is the anchor sample, is a positive sample, and both belong to the positive sample pair; is the second negative sample set; is the preset temperature coefficient; is the union of the positive sample and the second negative sample set. The loss function is marked as the loss function value calculated after each iteration, the current number of iterations is obtained, and the loss function value and the number of iterations are summarized to obtain the calculation result.

[0054] The pre-trained network model is iteratively adjusted according to the calculation results. As mentioned above, the pre-trained network model includes a first sub-network and a second sub-network, and the two sub-networks adopt different update and parameter adjustment processes. The parameters of the first sub-network are gradient updated; the parameters of the second sub-network are momentum updated. The specific update process can be: (2) In the above formula, are the parameters of the first sub-network, are the parameters of the second sub-network, is the preset hyperparameter, It can usually be set to 0.999, or other values. It can be set as needed. This is just for illustration and not for limitation.

[0055] In one embodiment, after each round of iteration, a pseudo-label of the sample set is obtained; based on the pseudo-label, the negative samples are stored in the sub-team corresponding to the feature queue; during the current round of iterative training, the negative samples and simulated defect images in the feature queue obtained in the previous round are selected as inputs of the pre-trained network model for processing; and the negative samples processed in the current round are used to update the feature queue.

[0056] In one embodiment, during the current round of iterative training, the feature vectors stored in the feature queue of the previous round are selected; if it is the first round, the feature vectors obtained by simulating defect image processing are selected; and / or both are used as the training data of this round as the input of the pre-trained network model for training. And after each round of iterative update, the feature vectors obtained in each round are stored in the feature queue through pseudo labels. By utilizing the characteristics of the feature queue dequeuing and queuing, and the sub-team only storing the feature vectors of the preset upper limit, it is ensured that the samples stored in the feature queue in the form of feature queue are the latest.

[0057] Furthermore, the above descriptions are all for the pre-training phase of the pre-trained network model. The end iteration condition of the pre-training phase is to determine whether the calculation result meets the first training condition. Among them, the calculation result includes the loss function value of the current round and the number of iterations of the current round. The loss function value is calculated by the loss function shown in formula (1). The first training condition includes: the loss function value is less than or equal to the preset acceptable threshold; and / or the number of iterations is greater than or equal to the preset maximum number of iterations. If the first training condition is not met, the iterative training continues as described above; if it is met, the last round of pre-trained network model is marked as the first model for the subsequent task fine-tuning phase.

[0058] Step S140: obtain a second model based on the first model; obtain a set of real defect images, and use the set of real defect images as the second model input to train the second model; when the second model meets the second training condition, mark the second model as an industrial defect detection model.

[0059] In one embodiment, a second model is obtained by modifying the first model, including: obtaining a backbone network and training weights of the first model; using the backbone network as an encoder of the second model, and initializing the encoder according to the training weights; constructing a classification head of the second model, the classification head adopts a fully connected layer structure with an input of 512 and an output of 6; and initializing the classification head with a preset mean and variance; a network model obtained by constructing the initialized encoder and classification head is marked as a second model; training the second model, including: inputting a real defect image into the second model for processing, obtaining the processed detection result, and all images in the real defect image set have labels; calculating the accuracy of the second model in the current round according to the detection results and labels; and adjusting the classification head of the second model according to the accuracy training.

[0060] In one embodiment, the second model is obtained by transforming the first model. Specifically, the second model can directly use the backbone network of the first model in the pre-training stage of step S130 as the encoder of the second model. Specifically, the encoder of the second model can be as follows: Figure 2 The backbone network of the first sub-network is shown. The encoder of the second model is initialized with the training weights, where the training weights are the weights learned by the backbone network of the first sub-network in the pre-training stage, and the encoder is initialized, that is, the convolutional layer of the second model is initialized.

[0061] At the same time, the classification head of the second model is constructed. Specifically, the classification head can use a fully connected layer with an input of 512 and an output of 6. The classification head can be initialized with a preset mean and variance, for example, a random number with a mean of 0 and a variance of 1.

[0062] The second model is further trained using a set of real defect images, where the real defect images are different from the simulated defect images. The real defect images are images of industrial defects that are actually collected under actual circumstances, and each real defect image is labeled, which indicates the actual industrial defect type of the corresponding real defect image. The real defect image set is used as the input of the second model and iterative training has been performed. Specifically, it can be divided into a training set and a test set according to a preset ratio, such as a ratio of 7:3, and the second model is trained through the training set to fine-tune the parameters of the second model. Among them, since the real defect images are all labeled, a fully supervised method can be adopted for training; it can also be trained with reference to the self-supervised learning method provided in the previous text application without limitation. The trained second model uses the test set as input to verify whether the training of the second model meets the corresponding requirements, and obtain the detection results obtained by the second model processing. The detection results refer to the training method in the previous text, and can include the second model determining the defect type of the real defect image classification in the test set. The accuracy of the second model is determined by the defect type and the label of the real defect image, and whether the training is qualified is determined according to the accuracy. The second training condition may specifically be that the accuracy is greater than a preset accuracy threshold, or that the number of iterations is greater than a preset upper limit.

[0063] If the second training condition is not met, the parameters can be adjusted iteratively until the second training condition is met. During the iterative parameter adjustment process, the parameters of the second model can be fine-tuned using the accuracy obtained from the current round of training. The parameter adjustment method for the pre-trained network model described above can also be referred to, and there is no restriction on this.

[0064] However, it is worth noting that for the training process of the second model, this application uses a linear evaluation method to verify the effectiveness of the method, that is, freezing the convolution layer of the second model and only training the classification head of the second model. If the second training condition is met, it means that the fine-tuned second model can complete the classification of industrial defects, and the second model is marked as an industrial defect detection model, and the training ends.

[0065] Inspired by the class conflicts in the random sampling of negative samples in traditional instance discrimination class contrast learning and the problem of low quality of negative samples caused by random sampling, the training method of this application aims to alleviate the class conflict problem in the instance discrimination process through defect type queues. When sampling negative samples, samples that belong to different categories from the anchor points will be selected, which can help the model extract more discriminative features. In addition, the input of the model is also different from the traditional industrial defect analysis method. By adding noise to the input image to simulate specific industrial defect categories, the model is guided to focus on different defect types. After pre-training on a large-scale industrial image dataset, this method can be applied to downstream industrial defect classification tasks through transfer learning or feature extraction, thereby improving the performance of downstream classification tasks.

[0066] After the training method described above, an industrial defect detection model that can be used for industrial defect classification can be obtained. Regarding the specific use of the industrial defect detection model, the present application also provides an industrial defect detection method, including steps S210 to S220.

[0067] Step S210: Acquire an industrial defect detection model.

[0068] Step S220: obtaining an image to be detected as an input of the industrial defect detection model, and obtaining a classification result output by the industrial defect detection model after being processed by the industrial defect detection model.

[0069] For the industrial defect detection model trained in step S210, the training method described in the previous steps S110 to S130 can be referred to, and will not be repeated here. Then, the image to be detected that needs to be detected and classified is input into the industrial defect detection model for processing, and the industrial defect detection model will output a classification result, which is used to indicate one or more possible industrial defect types of the image to be detected.

[0070] Therefore, this application obtains industrial simulation images as training data, and then guides the model to focus on different defect types, so as to more accurately learn the characteristics of industrial images with different defects, which is helpful for accurate classification. In addition, by eliminating potential positive samples, the problem of low quality of negative samples caused by class conflicts and random sampling is alleviated. And by using pre-training of simulated images, and transfer learning or feature extraction after pre-training to apply to the classification task of real industrial defect images, the performance of downstream classification tasks is improved through further model fine-tuning, and the accuracy of classification is improved.

[0071] The present application also provides an industrial robot, comprising a processor and a memory: the processor is used to execute a computer program stored in the memory to implement the above-mentioned industrial defect detection method. Figure 5, shows the internal structure of an industrial robot in one embodiment. Figure 5 As shown, the industrial robot includes a processor, a memory and a network interface connected via a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the industrial robot stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor may implement an industrial defect detection model training method and / or an industrial defect detection method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor may execute an industrial defect detection model training method and / or an industrial defect detection method. Those skilled in the art will understand that Figure 5 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the industrial robot to which the scheme of the present application is applied. A specific industrial robot may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0072] In one embodiment, the industrial robot may also include an I / O interface and an image acquisition module. The I / O interface is used to interact with external devices. For example, in the embodiments cited above, the industrial robot is a part of the unmanned factory system, so the industrial robot can be connected to the unmanned factory system through the I / O interface to be controlled by the system as a whole, so as to realize the unmanned and intelligent production chain. Furthermore, the industrial robot may also include an image acquisition module, which can be used to obtain the image to be detected to implement the industrial defect detection method provided in the embodiments of the present application. In other embodiments, the image to be detected can also be obtained through the I / O interface, and there is no restriction on the acquisition method.

[0073] In one embodiment, the application scenario of the industrial robot of the present application may be suitable for use in unmanned factories. Because the industrial robot is equipped with an industrial defect detection method, it can have a certain degree of intelligence, that is, it can automatically detect whether the relevant parts have industrial defects according to the incoming materials, thereby realizing automatic screening of materials.

[0074] Taking the application scenario of an unmanned factory as an example, the intelligence of the industrial robot provided in the embodiment of the present application is specifically reflected in four dimensions: robot perception, robot decision-making, management platform and control platform, and tool chain.

[0075] 1. Robot perception. In unmanned factories, industrial robots equipped with industrial defect detection methods have highly accurate perception capabilities. Through advanced machine vision systems, robots can capture images of products on the production line in real time, and use high-resolution image sensors installed in the robots, as well as optimized light sources and lighting systems to ensure high quality and stability of images. The robot can automatically perform image preprocessing, including denoising, enhancement and restoration, as well as image segmentation, to accurately extract the images to be detected required by the industrial defect detection model, and use the images to be detected as input to the model, providing a reliable data basis for subsequent defect detection.

[0076] 2. Robot decision-making. After acquiring the image to be detected, the industrial robot can make intelligent decisions using the built-in industrial defect detection method. These algorithms are based on advanced technologies such as deep learning and machine learning, and can automatically locate, identify and classify defects on the surface of the product. The training process of the industrial defect detection model required for the industrial defect detection method and the actual processing process have been described in detail in the previous embodiments, so they will not be repeated here. The robot can judge whether the product is qualified based on the classification results output by the industrial defect detection model, and make corresponding processing decisions in real time, such as continuing to transport qualified products to the next process, or removing defective products from the waste collection area. In addition, the robot can also continuously optimize its decision-making ability based on historical detection data and real-time feedback to improve the accuracy and efficiency of detection.

[0077] 3. Management platform and control platform. In order to achieve comprehensive monitoring and management of industrial robots, unmanned factories are usually equipped with advanced management platforms and control platforms. That is to say, as a large system, unmanned factories, industrial robots and the management and control platforms on which they are located are all part of the entire large system. Through the intelligent decision-making and deployment of the system, the entire factory can operate on its own without the participation of personnel, thereby improving efficiency. For the management and control platforms where the industrial robots are located, the management platform can display the robot's operating status, detection data, production progress and other information in real time, provide managers with an intuitive monitoring interface, or provide more detailed operating data for the unmanned factory system, so as to facilitate automatic deployment of the unmanned factory system. The control platform is responsible for receiving instructions from managers or unmanned factory systems, and making real-time adjustments and optimizations to the robot's operating parameters, detection strategies, etc. Through these two platforms, managers or unmanned factory systems can remotely monitor, diagnose and eliminate faults of industrial robots, as well as flexibly adjust production plans to ensure stable operation and efficient output of the production line.

[0078] 4. Tool chain. As mentioned above, the robot equipped with the industrial defect detection method is only one part of the entire factory system. In the entire system of the unmanned factory, the tool chain of the industrial robot includes multiple links such as data acquisition tools, storage management technology, data preprocessing, model training, model deployment and data labeling. The data acquisition tool is responsible for capturing the product image data on the production line in real time; the storage management technology ensures the safe and reliable storage of this data; data preprocessing can be used to extract useful information and reduce data redundancy, as well as further improve the accuracy and availability of data. The model training and deployment links are the core of the industrial defect detection method. By continuously iterating and optimizing the model, the accuracy and generalization ability of defect detection can be improved. Data labeling is used to provide rich labeled data for model training, supporting the continuous improvement and optimization of the model. These tool chain links together constitute the complete technical system for industrial robots to implement industrial defect detection methods.

[0079] As mentioned above, the industrial robot equipped with the industrial defect detection method provided by this application has certain intelligence. According to the deployment of the management or control platform, it is organically integrated into the system of the entire unmanned factory, which can further provide intelligence and realize more intelligent application scenarios.

[0080] In one embodiment, the present application further proposes a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the method described in any of the aforementioned embodiments.

[0081] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0082] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for training an industrial defect detection model, characterized in that: The steps include: Acquire a simulated defect image, and use the simulated defect image as an input of a pre-built pre-trained network model for processing; Obtaining a sample set output by the pre-trained network model, the sample set comprising positive sample pairs and a first negative sample set; performing a potential positive sample elimination operation on the first negative sample set to obtain a second negative sample set; The positive sample pair and the second negative sample set are used as inputs of a loss function for processing, and a calculation result of the loss function calculation output is obtained to iteratively adjust the pre-trained network model until a first training condition is satisfied; and the pre-trained network model that satisfies the first training condition is marked as a first model; A second model is obtained by transforming the first model; a set of real defect images is obtained and the set of real defect images is used as the input of the second model to train the second model; when the second model meets the second training condition, the second model is marked as an industrial defect detection model.

2. The industrial defect detection model training method according to claim 1, characterized in that: The step of acquiring a simulated defect image comprises: Acquire an input image, where the input image is a captured image of a real mechanical product; Randomly acquiring at least one simulated noise from an industrial defect category library, each of the simulated noises corresponding to an industrial defect, wherein the industrial defects include dimensional deviation, surface defect, assembly problem and material defect; At least one of the simulated noises is added to the input image through random data enhancement to obtain a pair of simulated defect sub-images, and all pairs of the simulated defect sub-images are aggregated to obtain the simulated defect image.

3. The industrial defect detection model training method according to claim 1, characterized in that: The simulated defect image includes a first simulated defect sub-image and a second simulated defect sub-image, and the first simulated defect sub-image and the second simulated defect sub-image are obtained by processing one image; The pre-trained network model includes a first sub-network and a second sub-network, wherein the first sub-network and the second sub-network have the same network structure, including a backbone network and a projection head; The step of processing the simulated defect image as an input of a pre-built pre-trained network model comprises: The first sub-network takes the simulated defect first sub-graph as input for processing, and the second sub-network takes the simulated defect second sub-graph as input for processing; The backbone networks of the first sub-network and the second sub-network encode the input simulated defect image into a latent representation; The projection heads of the first sub-network and the second sub-network project the potential representation into a feature vector, and characterize the sample set in the form of the feature vector.

4. The industrial defect detection model training method according to claim 3, characterized in that: The obtaining of the sample set output by the pre-trained network model comprises: The feature vector output by the first sub-network is called an anchor sample, and the feature vector output by the second sub-network is called a positive sample. A pair of the anchor sample and the positive sample is regarded as a group of positive sample pairs, and the anchor sample and the positive sample attract each other in the sample space; Mark all positive samples except the positive sample itself as negative samples corresponding to the positive sample, and aggregate all negative samples to obtain a first negative sample set corresponding to the positive sample; All the positive sample pairs and the corresponding first negative sample sets are aggregated to obtain the sample set.

5. The industrial defect detection model training method according to claim 1, characterized in that: After performing the potential positive sample elimination operation on the first negative sample set, a second negative sample set is obtained, including: The sample set has a pseudo label, and the pseudo label is used to instruct the pre-trained network model to classify and identify the industrial defect corresponding to the industrial defect simulation image; According to the pseudo-label, the negative samples in the first negative sample set are stored in a feature queue; the feature queue includes multiple columns of sub-queues, each column of the sub-queue corresponds to a pseudo-label, and each column of the sub-queue stores a preset number of negative samples; For the first negative sample set corresponding to each positive sample, the negative samples in the same sub-team as the positive sample are removed from the first negative sample set, and the remaining negative samples are marked as the second negative sample set; the second negative sample and the positive sample pair are mutually exclusive in the sample space.

6. The industrial defect detection model training method according to claim 1, characterized in that: The pre-trained network model includes a first sub-network and a second sub-network, and the positive sample pair includes an anchor sample and a positive sample; The processing of the positive sample pair and the second negative sample set as inputs of a loss function, obtaining a calculation result of the loss function calculation output and iteratively adjusting the pre-trained network model until the first training condition is met includes: The loss function is expressed as: ； In the above formula, is the anchor sample, is the positive sample, is the second negative sample set, is the preset temperature coefficient; Marking the loss function calculated after each iteration as a loss function value, obtaining the current number of iterations, and summing up the loss function value and the number of iterations to obtain the calculation result; According to the calculation result, the parameters of the first sub-network are updated by gradient, and the parameters of the second sub-network are updated by momentum. The updating process is: ； In the above formula, are the parameters of the second sub-network, are the parameters of the first sub-network, are the preset hyperparameters; The first training condition includes: The loss function value is less than or equal to a preset acceptable threshold; and / or, The number of iterations is greater than or equal to a preset maximum number of iterations.

7. The industrial defect detection model training method according to claim 6, characterized in that: The step of obtaining the calculation result of the loss function calculation output and iteratively adjusting the pre-trained network model comprises: After each round of iteration, a pseudo label of the sample set is obtained; according to the pseudo label, the negative sample is stored in the sub-queue corresponding to the feature queue; During the current round of iterative training, the negative samples in the feature queue obtained in the previous round and the simulated defect images are selected as the input of the pre-trained network model for processing; and the negative samples obtained in the current round of processing are used to update the feature queue.

8. The industrial defect detection model training method according to claim 1, characterized in that: The step of transforming the first model into a second model comprises: Acquire a backbone network and training weights of the first model; use the backbone network as an encoder of the second model, and initialize the encoder according to the training weights; Constructing the second model classification head, the classification head adopts a fully connected layer structure with an input of 512 and an output of 6; and initializing the classification head using a preset mean and variance; A network model constructed according to the initialized encoder and the classification head is marked as the second model; The training of the second model comprises: After the real defect image is input into the second model for processing, a detection result obtained by processing is obtained, and all images in the real defect image set have labels; Calculate the accuracy of the second model in the current round according to the detection result and the label; The classification head of the second model is adjusted according to the accuracy training.

9. An industrial defect detection method, characterized in that: include: The industrial defect detection model is obtained by training according to the method according to any one of claims 1 to 8; An image to be detected is obtained as an input of the industrial defect detection model, and after being processed by the industrial defect detection model, a classification result output by the industrial defect detection model is obtained.

10. An industrial robot, characterized in that: including a processor and a memory; The processor is configured to execute the computer program stored in the memory to implement the method according to claim 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 or the method according to claim 9 is implemented.

Images