A method and apparatus for sustainable learning of multiple industrial defect classification

Abstract

The application provides a sustainable learning multi-industrial defect classification method and device, and belongs to the technical field of defect classification and the technical field of sustainable learning. The method comprises the following steps: collecting an image of a product to be classified at random, and marking a corresponding classification task serial number on the image; inputting the image and the classification task serial number into a preset multi-task defect classification network; and outputting a classification result of whether the image belongs to a defect image under the classification task by the multi-task defect classification network. The application can realize sustainable and effective classification of multiple defects in an industrial scene, is suitable for complex defect classification tasks, and significantly improves the efficiency of defect classification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of defect classification and the technical field of continual learning, and particularly relates to a multi-industrial defect classification method and device capable of continual learning. BACKGROUND

[0002] At present, the production of industrial products in China relies on mechanical automatic manufacturing. Due to factors such as process and operation, defects are inevitable. The types of different defects are complex, and the differences between the same defects are large, which brings great difficulty to detection. The complexity of different types of defects mainly reflects in three aspects. First, the difference between classes is large. The appearance defects of industrial products are complex and diverse. The morphological characteristics between different types of defects may be very different. This difference leads to poor universality of the detection algorithm. Many defects need to be developed separately. The development complexity is extremely high. Second, the class boundary is large. The class boundary is another extreme of the large difference between classes. The apparent characteristics of different types of defects have certain similarity, and it is difficult to distinguish the types of defects, so it is also difficult to judge the causes of the defects. Third, the background is complex. In the production scene, it is difficult to completely separate the defects and the background, and the defect features are not obvious. In addition, the order of occurrence of different types of defects in the industrial scene is not certain. Therefore, it is necessary to study a multi-industrial defect classification method capable of continual learning, to realize the continual classification of multiple types of defects and to be less affected by the training order of defect types, so as to help enterprises improve product quality and reduce safety accidents caused by defective products.

[0003] Although human eye classification can identify product defects according to relevant standards and has strong adaptability, due to the limited working time of human beings, visual fatigue is easy to occur, and with the passage of time, the correct rate of classification will fluctuate or even decrease significantly. Detection equipment or detection models relying on machine vision and deep learning technology can maintain high detection accuracy for a long time, but the adaptability is weak. When new types of defect data appear, the model will greatly decrease the detection accuracy of old types of defect data in order to fit the new types of defect data, resulting in catastrophic forgetting. The most original solution to catastrophic forgetting is to retrain the network using all known data, but this will greatly consume time and space costs. The continual learning method based on playback needs to save a part of old data for training, but due to memory limitations or privacy issues, old data is sometimes difficult to obtain, thereby limiting the application field of the algorithm. The continual learning method based on parameter isolation allocates different model parameters for each task to prevent any possible forgetting, but the size of the model will increase linearly with the increase of the number of tasks. The continual learning method based on regularization prevents forgetting of old data by limiting large changes in model parameters, without the need to save old data and without a large increase in model parameters, but the current method still has a certain degree of forgetting with the increase of the number of tasks. SUMMARY

[0004] The application aims to overcome the problem that the current industrial detection algorithm has poor classification effect on multiple defects due to large differences in defect categories, and proposes a sustainable learning multiple industrial defect classification method and device. The application can realize continuous and effective classification of multiple defects in an industrial scene, is suitable for complex defect classification tasks, and significantly improves the efficiency of defect classification.

[0005] The first aspect embodiment of the application proposes a sustainable learning multiple industrial defect classification method, comprising:

[0006] Any product image to be classified is collected, and the image is labeled with a corresponding classification task serial number;

[0007] The image and the classification task serial number are input into a preset multi-task defect classification network, and the multi-task defect classification network outputs a classification result of whether the image belongs to a defect image under the classification task.

[0008] In one specific embodiment of the application, before the image and the classification task serial number are input into the preset multi-task defect classification network, the method further comprises:

[0009] The multi-task defect classification network is trained;

[0010] The training of the multi-task defect classification network comprises:

[0011] 1) Obtain normal images and defect images of different categories of products, wherein the normal images and the defect images are consistent in size with the product image to be classified;

[0012] 2) Group the normal images and the defect images under each category into a task data set, and label the task data set with a corresponding classification task serial number;

[0013] 3) Construct a multi-task defect classification network:

[0014] 4) Randomly sort all task data sets, and train the multi-task defect classification network with each task data set in turn according to the sorting to obtain a trained multi-task defect classification network.

[0015] In one specific embodiment of the application, the method further comprises:

[0016] The defect images in any task data set are expanded through data augmentation technology, so that the ratio of the number of defect images to the number of normal images in the task data set is close to 1:1.

[0017] In one specific embodiment of the present application, the backbone network of the multi-task defect classification network is a resnet18 network, and a feature encoding layer FEL is added after each convolution layer, each convolution group, each convolution layer in each convolution group, and the average pooling layer of the backbone network.

[0018] In one specific embodiment of the present application, the feature encoding layer is used to generate a corresponding permutation matrix according to the classification task sequence number, and the feature map generated by the task data set when passing through the multi-task defect classification network is rearranged according to the permutation matrix.

[0019] wherein, for the task data set with the classification task sequence number k, the permutation matrix generated by the mth feature encoding layer is:

[0020] S m (k) : = random permutation of I m with seed(k)

[0021] wherein, m = 1, 2, …, M, M is the total number of feature encoding layers in the multi-task defect classification network; the original matrix I m of the mth feature encoding layer is a unit matrix, and the row and column number is equal to the size of the last dimension of the feature map matrix; seed(k) represents the permutation order generated by taking the classification task sequence number k as a seed.

[0022] In one specific embodiment of the present application, the experience loss function of the multi-task defect classification network during training is:

[0023]

[0024] wherein, the K ordered task data sets are denoted as supervised learning tasks {T k |k∈{1,2,3,…K}} that arrive in order, T k represents the kth task data set in order, and K is the total number of task data sets; the number of images in the task data set T k is denoted as n k , x k,i represents the ith image in the kth task data set in order, y k,i represents the corresponding class label of the image x k,i , i.e. belongs to a defect image or a normal image; L k (θ) is the experience loss function during training of the kth task data set in order, θ is the parameter of the multi-task defect classification network, θ = {θ1, θ2, θ3, … θ L}, wherein θ iparameters of the i-th layer of the multi-task defect classification network, i = 1, 2, …, L, L is the total number of layers of the multi-task defect classification network, and the image x k,i The loss function of the image x k,i is expressed as l(f(θ; k, x k,i ).

[0025] In one specific embodiment of the present application, the multi-task defect classification network updates the parameters of the multi-task defect classification network by modifying the gradient during training, in the following manner:

[0026] When training the k-th task data set in the sorted order, the original gradient g l of the l-th layer of the multi-task defect classification network is calculated according to the following expression:

[0027]

[0028] The original gradient g l is modified using a positive definite symmetric matrix P, where the positive definite symmetric matrix P of the l-th layer is denoted as P l , and the expression is as follows:

[0029]

[0030] where H j,l is the Hessian matrix of the l-th layer when training the j-th task data set;

[0031] The modified gradient g of the l-th layer is:

[0032]

[0033] The parameters θ l of the l-th layer are updated according to the following formula:

[0034]

[0035] where η is the learning rate during network training;

[0036] The initial value of P l is set to the unit matrix I l ; after training each task data set, P l is updated according to the following formula:

[0037]

[0038] where, is an intermediate variable in the matrix inversion lemma

[0039]

[0040] The second aspect of the present application provides a device for classifying multiple industrial defects with sustainable learning, comprising:

[0041] An image acquisition module is configured to acquire an image of a product to be classified, and mark the image with a corresponding classification task sequence number;

[0042] A defect classification module is configured to input the image and the classification task sequence number into a preset multi-task defect classification network, and the multi-task defect classification network outputs a classification result of whether the image belongs to a defect image under the classification task.

[0043] The third aspect of the present application provides an electronic device, comprising:

[0044] at least one processor; and a memory connected to the at least one processor in communication;

[0045] The memory stores instructions executable by the at least one processor, and the instructions are configured to perform the above-mentioned method for classifying multiple industrial defects with sustainable learning.

[0046] The fourth aspect of the present application provides a computer readable storage medium, which stores computer instructions for executing the above-mentioned method for classifying multiple industrial defects with sustainable learning.

[0047] The present application has the following characteristics and advantages:

[0048] The present application is based on a continuous learning algorithm, which can realize continuous and effective classification of multiple defects in an industrial scene. When new class defects participate in training, it is not necessary to integrate all data for retraining, but only to mark a new task sequence number for the new class defects and train based on the existing model. While achieving high classification accuracy on new data, the classification accuracy on old data will not be significantly reduced.

[0049] The present application does not need to use the old data, and can ignore the problem that the old data cannot be obtained due to privacy or insufficient memory; the present application prevents forgetting by modifying the update direction of the gradient during back propagation, and does not need to allocate model parameters for each task separately, preventing the linear growth of model size due to the increase of the number of tasks.

[0050] The present application adds a feature encoding layer (FEL) in the network, each task has a specific feature arrangement method, thereby providing an effective method to eliminate the interference between tasks, reduce forgetting, and reduce the influence of task order on model performance, which is also more consistent with the uncertainty of the occurrence time of different defects in the actual industrial scene.

[0051] The application only needs to train one network to realize continuous classification of various defects in an industrial scene, and the classification accuracy of the final network is close to that of a multi-task training model, which is suitable for complex defect classification tasks and significantly improves the efficiency of defect classification. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 A whole flowchart of the multi-industrial defect classification method of sustainable learning of an embodiment of the application.

[0053] Figure 2 A structure diagram of the multi-task defect classification network in one specific embodiment of the application. DETAILED DESCRIPTION

[0054] The application provides a multi-industrial defect classification method and device of sustainable learning, which is described in further detail below in combination with the drawings and specific embodiments.

[0055] An embodiment of the first aspect of the application provides a multi-industrial defect classification method of sustainable learning, comprising:

[0056] An image of a product to be classified is collected at random, and the image is labeled with a corresponding classification task serial number;

[0057] The image and the classification task serial number are input into a preset multi-task defect classification network, and the multi-task defect classification network outputs a classification result of whether the image belongs to a defect image under the classification task.

[0058] In one specific embodiment of the application, the multi-industrial defect classification method of sustainable learning has a whole flowchart as shown in Figure 1 The flowchart is divided into a training phase and a test phase, and comprises the following steps:

[0059] 1) Training phase;

[0060] 1-1) Obtain defect images and normal images of different categories;

[0061] Product image data is collected on an actual production line, including various defect images and normal images occurring in the production process of products. The collected images are manually classified and labeled to obtain the labeling results of the defect images and normal images under each category. In this embodiment, there are at least 80 defect images and normal images under each category.

[0062] In one specific embodiment of the present application, 102 transformer pin deformation images, 353 normal pin images, 84 transformer copper wire defect images, 306 normal copper wire images, 100 transformer twisted wire defect images, 350 normal twisted wire images, 600 concrete crack images, 600 normal concrete images, 295 coffee bean debris images, and 590 normal coffee bean images, and 300 paperboard damage images, and 300 normal paperboard images are collected.

[0063] 1-2) Task division and unbalanced data preprocessing;

[0064] The defect images and normal images in each category obtained in step 1-1) are used to form a task data set, and the corresponding classification task number is marked. In this embodiment, the classification task numbers of the transformer pin data set, the transformer copper wire data set, the transformer twisted wire data set, the concrete data set, the coffee bean data set, and the paperboard data set are 1, 2, 3, 4, 5, and 6, respectively. When training a data set, the corresponding task number also participates in network training. Each task is a binary classification problem, and there is no class intersection between different tasks. The number of images in different task data sets can be different.

[0065] It should be noted that in actual production, the number of normal images is much larger than the number of defect images. Such class imbalance greatly affects the accuracy of supervised classification methods, resulting in the model being unable to accurately classify images. Ordinary classifiers usually aim to minimize the overall training error, which makes the model focus on the class of the majority of samples during the training process and produces overfitting. For the minority samples, underfitting occurs because they are not specially considered. This results in the classification result being more favorable to normal images, and the generalization ability of the classifier is poor. To solve this problem, in the embodiment of the present application, for any task data set, the number ratio of normal images to defect images is required to be close to 1:1. If the number of defect images in any task data set is less than 200, data augmentation techniques (horizontal flip, vertical flip, brightness increase, contrast increase, and saturation increase) can be used for expansion. The size of all training images needs to be cropped consistently, and the size is not required.

[0066] 1-3) Constructing a multi-task defect classification network to ensure that a single task can achieve high defect classification accuracy after sufficient training;

[0067] Due to the complexity of different defects in industrial scenes and the large difference between the same defects, a deep network is needed to extract rich semantic features. On the other hand, to prevent network degradation caused by simply increasing the depth, resnet18 is finally selected as the backbone network of the multi-task defect classification network in this embodiment. In one specific embodiment of the present application, the structure of the multi-task defect classification network is as follows: Figure 2As shown in the figure, a feature encoding layer (FEL) is added after the convolution layer of the backbone network resnet18, each convolution group, each convolution layer in the convolution group, and the average pooling layer, respectively. In one specific embodiment of the application, the first convolution layer has a convolution kernel size of 7*7, an output channel of 64, and a step size of 1; followed by four convolution groups, each of which contains two residual blocks, and each residual block contains two convolution layers; in the first convolution group, the convolution kernel size is 3*3, the step size is 1, and the output channel is 64; in the second convolution group, the convolution kernel size is 3*3, the step size of the first convolution layer is 2, and the step size of the remaining three convolution layers is 1, and the output channel is 128; in the third convolution group, the convolution kernel size is 3*3, the step size of the first convolution layer is 2, and the step size of the remaining three convolution layers is 1, and the output channel is 256; in the fourth convolution group, the convolution kernel size is 3*3, the step size of the first convolution layer is 2, and the step size of the remaining three convolution layers is 1, and the output channel is 512; followed by an average pooling layer and a fully connected layer.

[0068] Further, the embodiment additionally introduces a virtual feature encoding layer (FEL), which applies task-specific rearrangement to the input feature map internally, and the arrangement order is randomly generated using the task sequence number as a seed. For example, when training the first task data set, the task sequence number 1 is used as a seed to generate a specific arrangement order, and the feature maps of all input images are rearranged in the feature map according to this arrangement order. The size of the feature map input to the FEL and the feature map output by the FEL does not change. The FEL only rearranges the existing feature map, and the arrangement order corresponding to each FEL does not change during the training of the task data set (the order of the same FEL layer is determined when training a task data set, and the order of different FEL layers for the same task data set is also determined). Write the FEL in matrix form, and the arrangement matrix of the mth (m = 1, 2, …, M) virtual feature encoding layer is:

[0069] S m (k) : = random permutation of I m with seed(k)

[0070] The feature map matrix is multiplied by the arrangement matrix to obtain the rearranged matrix output by the FEL layer.

[0071] where I mis an identity matrix whose row and column number is equal to the size of the last dimension of the feature map matrix. In this embodiment, M = 22, i.e. there are 22 virtual feature encoding layers in total (in this embodiment, one feature encoding layer is added after each convolution layer of the backbone network resnet18, each convolution group, each convolution layer in the convolution group, and the average pooling layer, wherein there are 4 convolution groups in total, and each convolution group contains 4 convolution layers). seed(k) represents the permutation order generated by the classification task number k as a seed. Considering that the order of features is crucial for the next layer to obtain interpretable information, if there is no correct feature order, the recognition ability of the network will be greatly reduced. FEL provides an effective method to eliminate interference between tasks. Therefore, the same feature extractor (in this embodiment, the convolution kernel) plays different roles in different tasks.

[0072] During training, the input of the multi-task defect classification network is a batch of images and the corresponding task number in any task data set. In this embodiment, a batch of images is 10 images, forming a matrix with a size of [10, 3, 32, 32], wherein the first dimension is the batch size, the second dimension is the image channel number, and the third and fourth dimensions are the height and width of the image. The output of the first convolution layer of the multi-task defect classification network is a matrix with a size of [10, 64, 32, 32], the output of the first convolution block is a matrix with a size of [10, 64, 32, 32], the output of the second convolution block is a matrix with a size of [10, 128, 16, 16], the output of the third convolution block is a matrix with a size of [10, 256, 8, 8], the output of the fourth convolution block is a matrix with a size of [10, 512, 4, 4], and the output of the average pooling layer is a matrix with a size of [10, 512, 1, 1]. The final output of the network is the defect classification result corresponding to the batch of images. In this embodiment, there are 6 task data sets in total, each containing 2 classes of normal images and defect images, i.e. 12 classes in total. Therefore, the output is a matrix with a size of [10, 12], wherein 10 is the batch size and 12 is the total number of image classes. Since the task number is specified during training, the network can finally determine whether the input image belongs to the normal image or the defect image of a specific task. Taking 1 defect image of the first task data set as an example, the output matrix of the network is [0.7, 0.2, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.01, 0.03], wherein each number is the probability score of the image belonging to each class. Since the task number is 1, the probability scores of other task data sets are set to be infinitesimal, and the matrix becomes [0.7, 0.2, -1e32, -1e32, -1e32, -1e32, -1e32, -1e32, -1e32, -1e32]. Since 0.7 is greater than 0.2, it can be determined that the image belongs to class 1, i.e. the defect image of the first task data set.

[0073] 1-4) using a continuous learning method, training the defect classification network established in step 1-3) using different task data sets to obtain a trained defect classification network;

[0074] In this embodiment, all task data sets are randomly sorted, and then each data set is used in turn to train the multi-task defect classification network according to the sorting. Each task data set can end training when it reaches the set upper limit of the same number of training rounds, and the number of rounds can be determined by itself. In this embodiment, the number of rounds is set to 50. After all task data sets are trained, the final trained defect classification network is obtained.

[0075] In one specific embodiment of the present application, K sorted task data sets are denoted as supervised learning tasks {T k |k∈{1,2,3,…K}}, where T k represents the kth sorted task data set, and K is the total number of task data sets, which is equal to 6 in this embodiment. In each task data set T k , there are n k images, x k,i represents the i-th image in the k-th sorted task data set, and y k,i represents the corresponding class label of the image x k,i , i.e. whether it belongs to a defect image or a normal image. θ is the parameter of the multi-task defect classification network, θ = {θ1, θ2, θ3, … θ L}, where θ i is the parameter of the i-th layer of the classification network, i = 1, 2, …, L, and L is the total number of layers of the multi-task defect classification network. The loss function of the image x k,i is represented as l(f(θ; k, x k,i ), y k,i ), and the empirical loss function of the task data set T k is defined as:

[0076]

[0077] The multi-task defect classification network has a total of L layers (the network of this embodiment has a total of 18 layers as shown in Figure 2 , including 17 convolutional layers and one fully connected layer, and FEL is not included), so when training the kth sorted task data set, the gradient g l of the lth (l = 1, 2, …, L) layer can be calculated as follows:

[0078]

[0079] If only task T kThen the gradient can be calculated according to the above formula to update θ. However, this method will encourage the neural network to gradually forget the old task. Therefore, the updating direction of the gradient is modified in the embodiment. For this purpose, the embodiment introduces a positive definite symmetric matrix P to modify the original gradient g l . The positive definite symmetric matrix P of the l-th (l = 1, 2, …, L) layer is denoted as P l :

[0080]

[0081] where H j,l is the Hessian matrix of the l-th layer when training the j-th task data set.

[0082] If P l is directly calculated according to the above formula, the solving process of the inverse matrix is complex, so the updating of P l is performed according to the matrix inversion lemma in the embodiment. P l is different in different tasks. The initial value of P l is set as the unit matrix I l ; after the training of each task data set is completed, the initial P l of the new task data set inherits the value at the end of the training of the previous task data set and is updated according to the following process:

[0083]

[0084]

[0085] where, is an intermediate variable of the matrix inversion lemma and has no actual meaning, P l is updated once after the training of a task data set is completed.

[0086] Then the modified gradient of the l-th layer is g

[0087]

[0088] The parameters θ l of the l-th layer can be updated according to the following formula:

[0089]

[0090] where η is the learning rate when training the network, and η = 0.0003 in the embodiment.

[0091] After the training of all task data sets is completed, the final trained multi-task defect classification network is obtained.

[0092] It should be noted that the six task data sets in the embodiment are trained in turn, and a preliminary multi-task defect classification network is obtained after the first task data set is trained, and the preliminary multi-task defect classification network is used as the current multi-task defect classification network, the second task data set is used as a new category defect data set and is input into the current multi-task defect classification network for training, and the process is recursively performed until the six task data sets are trained to obtain a final multi-task defect classification network. Based on the continuous learning algorithm, the continuous and effective classification of multiple defects in an industrial scene can be realized. When a new category defect participates in training, it is not necessary to integrate all data and retrain, but only to mark a new task sequence number for the new category defect and train based on the existing model. When the new data reaches a high classification accuracy, the classification accuracy on the old data will not be significantly reduced.

[0093] 2) test phase;

[0094] 2-1) randomly collecting a product image on the same production line as in the training phase;

[0095] 2-2) cropping the product image collected in 2-1) to the same size as the training phase image and marking the task sequence number;

[0096] 2-3) inputting the image in step 2-2) (in this embodiment, the image is one of a transformer pin, a transformer copper wire, a transformer twisted wire, a concrete crack, a coffee bean or a paperboard) and the corresponding task sequence number into the multi-task defect classification network trained in step 1), and the network outputs the classification result of the image; wherein if the image is a defect image, the output result is the category to which the defect image belongs; if the image is a normal image, the output is the category to which the normal image belongs.

[0097] To achieve the above embodiment, the second aspect embodiment of the present application proposes a sustainable learning multi-industrial defect classification device, comprising:

[0098] An image acquisition module is configured to randomly collect a product image to be classified, and mark the corresponding classification task sequence number on the image;

[0099] A defect classification module is configured to input the image and the classification task sequence number into a preset multi-task defect classification network, and the multi-task defect classification network outputs a classification result of whether the image belongs to a defect image under the classification task.

[0100] It should be noted that the foregoing embodiment of the method for classifying multiple industrial defects by sustainable learning is also applicable to the device for classifying multiple industrial defects by sustainable learning, and details are not repeated herein. The device for classifying multiple industrial defects by sustainable learning according to the embodiment of the present application can realize continuous and effective classification of multiple defects in an industrial scene, is suitable for complex defect classification tasks, and significantly improves the efficiency of defect classification.

[0101] To achieve the above-mentioned embodiments, the third aspect of the present application provides an electronic device, comprising:

[0102] at least one processor; and a memory connected with the at least one processor in communication;

[0103] wherein the memory stores instructions executable by the at least one processor, and the instructions are configured to perform the above-mentioned method for classifying multiple industrial defects by sustainable learning.

[0104] To achieve the above-mentioned embodiments, the fourth aspect of the present application provides a computer readable storage medium, which stores computer instructions for causing the computer to perform the above-mentioned method for classifying multiple industrial defects by sustainable learning.

[0105] It is noted that the computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wire, cable, RF, etc., or any suitable combination of the foregoing.

[0106] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device and be not assembled in the electronic device. The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method of the above-described embodiment for sustainable learning of various industrial defect classifications.

[0107] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0108] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0109] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0110] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the application include additional implementations in which the order of execution or the specific logic functions (or steps) can be changed, including according to the functionality involved, without departing from the scope of the embodiments of the application. It should be understood that the embodiments of the application can be practiced with additional combinations of hardware and software, and that the embodiments of the application can be implemented with hardware equivalent to software, with software equivalent to hardware, or with both, without departing from the scope of the embodiments of the application.

[0111] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of them. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, a portable computer diskette (magnetic), a RAM (random access memory), a ROM (read only memory), an EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM), a storage

[0112] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, can be used, or a combination thereof: discrete logic circuitry having logic gates for implementing logic functions upon an application of data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0113] Those of skill in the art would understand that the steps of the methods carried out above can be carried out by program instructions executed by relevant hardware, and the program can be stored in a computer readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0114] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0115] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A sustainable learning-based method for classifying various industrial defects, characterized in that, include: Collect any image of a product to be classified and label the image with the corresponding classification task number; The image and the classification task number are input into a preset multi-task defect classification network, and the multi-task defect classification network outputs the classification result of whether the image belongs to a defect image under the classification task. The empirical loss function of the multi-task defect classification network during training is: In the formula, the K sorted task datasets are denoted as the sequentially arrived supervised learning task {T}. k |k∈{1,2,3,…K}},T k Let T represent the k-th task dataset after sorting, where K is the total number of task datasets; Let T be the task dataset. k The number of images in the data is n k x k,i Let y represent the i-th image in the sorted k-th task dataset. k,i Representing the image x k,i The corresponding category label indicates whether it belongs to a defective image or a normal image; L k (θ) is the empirical loss function used when training the k-th task dataset after sorting, and θ is the parameter of the multi-task defect classification network, θ = {θ1, θ2, θ3, ..., θ}. L }, where θ i Let x be the parameters of the i-th layer of the multi-task defect classification network, i = 1, 2, ..., L, where L is the total number of layers in the multi-task defect classification network. k,i The loss function is expressed as l(f(θ;k,x)). k,i ),y k,i ); During training, the parameters of the multi-task defect classification network are updated by modifying the gradient, as follows: When training on the sorted k-th task dataset, the original gradient g of the l-th layer of the multi-task defect classification network is used. l The calculation is expressed as follows: Using the positive definite symmetric matrix P on the original gradient g l The modification is made, where the positive definite symmetric matrix P of the l-th layer is denoted as Pl. l The expression is as follows: Among them, H j,l It is the Hessian matrix of the l-th layer during training of the j-th task dataset; The modified gradient of the l-th layer for: Update the parameters θ of the l-th layer according to the following formula. l : Where η is the learning rate during network training; P l The initial value is set to the identity matrix I. l After training on each task dataset, P l Update according to the following formula: in, It is an intermediate variable in the matrix inversion lemma:

2. The method according to claim 1, characterized in that, Before inputting the image and the classification task number into a preset multi-task defect classification network, the method further includes: Train the multi-task defect classification network; Training the multi-task defect classification network includes: 1) Obtain normal images and defective images of different categories of products, wherein the normal images and defective images are the same size as the product images to be classified; 2) Combine the normal images and defective images under each category into a task dataset, and label the task dataset with the corresponding classification task number; 3) Construct a multi-task defect classification network: 4) Randomly sort all task datasets, and train the multi-task defect classification network sequentially using each task dataset according to the sorting to obtain the trained multi-task defect classification network.

3. The method according to claim 2, characterized in that, The method further includes: Defective images in any task dataset are augmented using data augmentation techniques so that the ratio of the number of defective images to the number of normal images in the task dataset is close to 1:

1.

4. The method according to claim 2, characterized in that, The backbone of the multi-task defect classification network is a ResNet18 network. A feature encoding layer (FEL) is added after each convolutional layer, each convolutional group, each convolutional layer in the convolutional group, and the average pooling layer of the backbone network.

5. The method according to claim 4, characterized in that, The feature encoding layer is used to generate a corresponding permutation matrix based on the classification task number, and to rearrange the feature maps generated when the task dataset passes through the multi-task defect classification network based on the permutation matrix; For the task dataset with classification task number k, the permutation matrix generated by the m-th feature encoding layer is: S m (k):=random permutation of I m with seed(k) In the formula, m = 1, 2, ..., M, where M is the total number of feature coding layers in the multi-task defect classification network; the original matrix I of the m-th feature coding layer... m It is an identity matrix whose number of rows and columns is equal to the size of the last dimension of the feature map matrix; seed(k) represents the order in which the classification task index k is used as the seed.

6. A multi-industry defect classification device based on the sustainable learning method of claim 1, characterized in that, include: The image acquisition module is used to acquire any image of a product to be classified and to label the image with the corresponding classification task number; The defect classification module is used to input the image and the classification task number into a preset multi-task defect classification network, and the multi-task defect classification network outputs the classification result of whether the image belongs to the defect image under the classification task.

7. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-5.

Images