Visual inspection method, system, storage medium, and device based on federated learning

Abstract

The present invention provides a visual inspection method based on federated learning, which overcomes the problem of limited industrial data set size in each inspection task; the model provided in the method can be applied across industries to ensure efficient deployment of the model in new industries that initially provide limited data, and the method can solve various problems in actual defect detection applications; the model in the server in the method can be continuously updated regularly and help each client upgrade its inspection model over time; in addition, the method requires the client to send data information obtained from the original image to the server through a deep learning model, and this information cannot be used to restore the original data, so there is no data security issue involved to ensure the data security of the client; the present invention also relates to a visual inspection system, storage medium, and electronic device based on federated learning.

Description

Technical Field

[0001] The present invention relates to the field of visual inspection technology, and in particular to a visual inspection method based on federated learning. Background Art

[0002] Appearance inspection of products is a common task for manufacturers in various industries. In China alone, there are more than 60 million workers doing this work. Machine vision has been used for a long time to improve product quality and reduce costs. Appearance inspection includes three main tasks: defect detection, presence detection, and measurement. Traditional automated vision technologies such as pattern matching have made great progress in measurement and presence detection. However, defect detection usually requires a lot of development work and expertise to design specialized algorithms for each type of surface or structural defect.

[0003] One of the biggest challenges in applying deep learning (DL) based methods to the industry is the lack of data samples for defect detection classification tasks. In fact, a common approach is to use the weights of the lower layers of a convolutional neural network (CNN) that has been pre-trained on a large dataset with an existing architecture (e.g., VGG, Inception, etc.) and retrain the top layers of the classifier on a dataset for a specific task. However, public models trained on a variety of image categories may not be sensitive enough for transfer learning for product surface defect inspection.

[0004] Another important issue about all traditional methods in automated surface inspection is that they all involve a modeler directly analyzing the original image dataset or a similar version of that dataset, exposing private information that manufacturers want to protect. In these cases, if the modeler is from an external source, data security is compromised, hampering the industry's growth. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a visual inspection method based on federated learning.

[0006] The present invention protects the customer's data privacy by training on the client side; this method enables new customers to be quickly deployed and establishes cross-industry transfer learning.

[0007] The present invention provides a visual inspection method based on federated learning, comprising the following steps:

[0008] Sending a request instruction to the server to obtain a first basic model;

[0009] Collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model;

[0010] The updated server-side model sent by the server is received at a set threshold time and trained to obtain a final model for performing the detection task for product appearance detection; wherein the final model is a model updated once or multiple times from the first basic model.

[0011] Preferably, in the step of obtaining the first basic model, the method further includes:

[0012] The collected regular data set is refined according to the set specifications and a convolutional neural network is trained to obtain the first basic model.

[0013] Preferably, the step of collecting defect image data to train the first basic model further includes:

[0014] The defect image data is tested using a pre-trained convolutional neural network model, and the average probability of the defect image data being classified into each class in the public dataset is obtained to obtain the differences between the defects in the defect image data and the degree of association with each class in the public dataset.

[0015] Preferably, the step of collecting defect image data to train the first basic model further includes:

[0016] Perform data augmentation on defect image data and generate affinity matrix;

[0017] The affinity matrix is ​​normalized by the analytic hierarchy process to obtain the maximum for all defect classes.

[0018] Preferably, the step of receiving the updated server-side model sent by the server within the set threshold time and performing training further includes:

[0019] The model training is performed using the first defect image data of the industry and the second defect image data in the public data set; wherein the first defect image data and the second defect image data are highly correlated.

[0020] Preferably, in the process of simultaneously performing model training on the first defect image data and the second defect image data, it further includes:

[0021] Performing data expansion and hierarchical analysis processing on the first defect image data and the second defect image data;

[0022] The normalized data of various industries in the processed public dataset are associated and appended together as an association matrix, and binary integer programming is used to obtain the final model.

[0023] The present invention also provides an electronic device, comprising: a processor; a memory; and a program, wherein the program is stored in the memory and is configured to be executed by the processor, and the program includes a method for executing a visual detection method based on federated learning.

[0024] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a visual detection method based on federated learning.

[0025] The present invention also provides a visual inspection system based on federated learning, including a request module, an update module, and a training module; wherein:

[0026] The request module is configured to send a request instruction to the server to obtain the first basic model;

[0027] The updating module is configured to collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model;

[0028] The training module is configured to receive the updated server-side model sent by the server at a set threshold time and perform training to obtain a final model for performing the detection task for product appearance detection; wherein the final model is a model that is updated once or multiple times from the first basic model.

[0029] Preferably, the updating module comprises a testing unit; the testing unit is configured to test the defect image data using a pre-trained convolutional neural network model, and obtain the average probability of the defect image data being classified into each class in the public data set, so as to obtain the difference between each defect between the defect image data and the degree of association with each class in the public data set; and perform data expansion on the defect image data and generate an affinity matrix;

[0030] The affinity matrix is ​​normalized by the analytic hierarchy process to obtain the maximum of all defect classes;

[0031] The training module includes a processing unit; the processing unit is configured to perform model training using first defect image data of the industry and second defect image data in a public data set; wherein the first defect image data and the second defect image data are highly correlated; and

[0032] Performing data expansion and hierarchical analysis processing on the first defect image data and the second defect image data;

[0033] The normalized data of various industries in the processed public dataset are associated and appended together as an association matrix, and binary integer programming is used to obtain the final model.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention discloses a visual inspection method based on federated learning, which overcomes the problem of limited scale of industrial data sets in each inspection task; the model provided in the method can be applied across industries to ensure efficient deployment of the model in new industries that initially provide limited data, and the method can solve various problems in actual defect detection applications; the model in the server in the method can be continuously updated regularly and help each client upgrade its inspection model over time; in addition, the method requires the client to send data information obtained from the original image to the server through a deep learning model, and this information cannot be used to restore the original data, so there is no data security issue involved to ensure the data security of the client; the present invention brings the security, reliability and efficiency in the field of automated inspection to a new level, and taps the huge potential of deep learning applications in this field.

[0036] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. The specific implementation of the present invention is given in detail by the following embodiments and their accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0038] Figure 1 is an overall flow chart of the visual inspection method based on federated learning of the present invention;

[0039] Figure 2 A logic diagram of an embodiment of a visual inspection method based on federated learning of the present invention;

[0040] Figure 3 A logic diagram of another embodiment of the visual detection method based on federated learning of the present invention;

[0041] Figure 4 is a schematic diagram of a specific embodiment of the visual inspection method based on federated learning of the present invention;

[0042] Figure 5 is a schematic diagram of another specific embodiment of the visual inspection method based on federated learning of the present invention;

[0043] Figure 6 It is an overall logic diagram of the visual detection method based on federated learning of the present invention;

[0044] Figure 7a A schematic diagram of a texture data set of a specific embodiment of the visual detection method based on federated learning of the present invention;

[0045] Figure 7b A schematic diagram of a wood data set of a specific embodiment of the visual inspection method based on federated learning of the present invention;

[0046] Figure 8 It is a module diagram of the visual inspection system based on federated learning of the present invention;

[0047] Description of the drawings: 10. Training module, 110. Processing unit, 20. Request module, 30. Update module, 310. Testing unit. DETAILED DESCRIPTION

[0048] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0049] The present invention provides a visual inspection method based on federated learning, which is a new attempt to solve the data dilemma faced by traditional machine learning methods. This is a new artificial intelligence basic technology that conducts efficient machine learning between multiple participants or multiple computing nodes under the premise of protecting data privacy and meeting legal and compliance requirements.

[0050] The present invention extends the provided method to establishing a cross-industry transfer learning basic model, rather than focusing on the defects of a single type of product or industry, which allows customers in emerging industries to quickly deploy basic models and perform acceptable automatic detection tasks, while service providers such as servers can continuously improve the model because the server can generate new data sets from the client.

[0051] Another important issue about all traditional methods in automatic surface inspection is that they all involve developers directly analyzing the original image dataset or a similar version of that dataset, exposing information that the manufacturer wishes to protect. In these cases, if the developer is from outside, the data cannot be touched due to data security considerations, and thus the model cannot be trained, which in turn hinders the development of product inspection technology in the industry. The present invention provides a solution, which is a computing framework based on the following method: installing a detection analysis network model on an edge detection device, and in FL, the detection model training is performed on the original device (edge) where the data is stored (such as IoT devices, mobile phones), rather than on a server. This allows the data to be retained on its end-user device. This is different from the current traditional way of training models, in which the device dataset must be passed to the server for training. In the present invention, we propose a novel federated analysis / learning framework for model training for automatic inspection to ensure customer data security.

[0052] In order to practically apply federated learning together with deep learning, which usually requires a large amount of data, there is a problem that the data (with labels) of each customer is scarce, which may pose a challenge to model training. This can be solved by the method disclosed in this paper.

[0053] The federated learning method provided by the present invention is applicable to appearance inspection, and the appearance inspection of products involves all walks of life. Specific federated learning methods, such as Figure 1 , Figure 6 As shown, the following steps are included:

[0054] S1. Send a request instruction to the server to obtain a first basic model;

[0055] S2, collecting defect image data to train the first basic model and extracting high-level features therein and sending them to the server to update the server-side model;

[0056] S3. Receive the updated server-side model sent by the server at a set threshold time and train it to obtain a final model for performing the inspection task for product appearance inspection; wherein the final model is a model updated once or multiple times with the first basic model. The server is an organization or individual that designs the model structure and provides automated inspection services to manufacturers. Each customer is one of the inspection sites in the manufacturer's production line that requires a high-quality algorithm to complete its specific inspection task. Some manufacturers may have hundreds of similar inspection sites, each of which is a client that communicates separately with the server to obtain and update its inspection model. It should be noted that each production line may have different inspection tasks, so each production line requires a different model.

[0057] Specifically, the client first requests the service, and the client sends a request to the server to obtain a new model for its inspection task, namely the first basic model, as well as some basic information about its data. The server will then analyze whether it is possible to finally obtain the inspection model or what approach to take. Typically, if the new client is a new industry for the server, or there are no sufficiently similar surface types that have been used in the server for the new client in previous cases, a general cross-industry base model will be provided for deployment directly as the initial solution for the production line. Alternatively, if the new customer is a similar production line to an old customer and has an easy-to-use custom model (updated from the previous cycle), the model will be provided to the new customer as a base model to train its available data and directly perform its surface inspection tasks with high accuracy.

[0058] like Figure 6 As shown, the manufacturer's existing inspection points (A, B, C, ...) in cycle i are already running their models. At this stage, they do not need any further action. Client X is a new installation on the production line, which first joins at some point in the deployment phase of cycle i. Client X needs to send a request to the server. The server approves it based on the client's data type and local computer settings, and responds by providing X with its initial base model. At any time during this stage of the cycle, preferably on the order of several months, if there is any other new client, it will also be given a base model after the request. Once the base model is received, the client will perform transfer learning locally using limited sample data and start the automated surface inspection process.

[0059] Typically, the deployment phase covers most of each cycle, with new clients joining at any time and deploying their initial models. However, for a manufacturer, from the beginning, a group of new clients join and will be provided with an initial base model at the same time. All clients need to collect new data during this phase to prepare for new model updates.

[0060] The server will periodically update its model and collect information from the running (e.g., real-time) clients. This will be done regularly on a certain schedule in each cycle, which can be once a year or once a quarter. The server will first send an algorithm and a new basic model to the client, which each client can use to train a preliminary model based on its newly collected defect image data, and then extract high-level features and send them back to the server. High-level features are features extracted from the high-level convolutional layers of the newly trained model. This will not leak the original image, so data privacy is still protected. If the client is not generating new image data at this time, there is no need to Figure 6 The characteristic data is sent back as in the case of client B in cycle i.

[0061] After the server receives all feature data from the client, it performs a set of operations, which may include the disclosed operations or other advanced methods, to update the new base model in the server with a greater weight on the new image features rather than the initial cross-industry model.

[0062] To be clear: even if a client has a small dataset (hundreds of images), it can still provide model updates with reasonable quality, because the updated base model can be applied to each client.

[0063] In this phase, the server will send the updated base model to all clients, and each client will train its new model locally with smaller original data with minimal time cost. Clients will then deploy their final model in their inspection system. After this short phase, the client and server will enter the deployment phase of the next cycle.

[0064] In the deployment of new customers, server updates and user-side updates, new customers can join at any time at all stages, while other customers focus on detection. The method provided by the present invention can be quickly deployed for each new customer, and most of the time in each cycle will be in the serious update stage, during which the cloud server will use huge computing power to train the basic model for the customer's final model. The client is encouraged to continuously collect data, which will help update the model on the server. The federated learning of the framework formed by this method is different from the federated learning in ordinary applications because the server in the framework provided by the present invention handles most of the calculations in the model update. A large amount of data flow of advanced functions will occur from the client to the server, while protecting the privacy of the data. All these aspects of the design are intended to provide manufacturers with fast model delivery and high-quality updates with minimal model upgrade time. The framework provided by the present invention uses an automatic inspection model of a convolutional neural network to continuously improve the basic model and solve the data scarcity and privacy of a single customer. On a wider scale, cross-industry models can also be updated through data collection from multiple industries. Ultimately, as data accumulates, cross-industry models will be trained on defect datasets instead of ImageNet (public datasets), thus becoming specialized basic models based on surface defects.

[0065] Step S2 also includes refining the collected conventional data set according to the set specifications and performing convolutional neural network training to obtain the first basic model. Figure 2As shown, the basic training process is disclosed in this embodiment. The method can be used to prepare a large data set for a basic model of a specific industry, and then can be used for transfer learning to obtain the final defect detection model. The method is used to help solve the difficulties caused by the lack of data on the local client performing deep learning training. The conventional data set is a publicly available data set. Figure 4 As shown, a model for ranking common classes for the wood dataset is shown, and how to rank common dataset classes in ImageNet for the wood surface defects dataset is given.

[0066] Step S2 also includes testing the defect image data using a pre-trained convolutional neural network model, and obtaining the average probability of the defect image data being classified into each class in the public data set, so as to obtain the difference between each defect between the defect image data and the degree of association with each class in the public data set. Data expansion is performed on the defect image data and an affinity matrix is ​​generated; the affinity matrix is ​​normalized by the hierarchical analysis method to obtain the maximum value of all defect classes. In this embodiment, Figure 3 As shown, the method itself involves using a pre-trained publicly available CNN model such as inception V3 to test the defect data set, and obtain the average probability that the defect data set is classified into each class in the public data set; then, the probability can be used in the data expansion process to quantify the difference between the defect classes in the defect data set and the degree of association between each class in the public data set. The generated affinity matrix can be normalized by a hierarchical analysis method, and the normalized affinity matrix is ​​used to finally obtain a global mapping scheme to maximize the performance of all defect classes while minimizing supervision, using binary integer programming (BIP) to solve the phenomenon selection problem. The method provided in this embodiment can use relatively small data samples to train classifiers for specific tasks. Unlike the typical method of adding multiple geometric transformations such as flipping, scaling, translation, etc. to the original image data to expand the number of samples in the training data set, this embodiment aims to quantify the relationship between different data sets and extract "structure" from them. "Structure" means a set of relationships that brake which data set or subset between data sets provides useful information to another, and how much information is provided.

[0067] like Figure 3As shown, step S3 also includes: using the first defect image data of the industry and the second defect image data in the public data set to jointly train the model; wherein the first defect image data and the second defect image data are highly correlated. Performing data expansion and hierarchical analysis processing on the first defect image data and the second defect image data; associating the normalized data of each industry in the processed public data set and attaching them together as an association matrix, and using binary integer programming to obtain the final model. In this embodiment, for the application of expanded industries, this embodiment discloses the need to create a basic model shared by defect data sets from different industries, specifically:

[0068] A cross-industry base model that can be used for transfer learning in the customer's local computer can be trained by a mixture of public dataset classes that are highly correlated with surface defect data for each industry. It should be understood that for each industry, there will be a set of dataset classes that are highly correlated with it, which are included in the selected datasets for base model training. Some selected classes may be highly correlated with multiple industries. The defect datasets of each industry are first data augmented and hierarchically analyzed separately, and then the normalized data associations from all industries are attached as an association matrix. Given this information, the BIP method under the cross-industry application budget strategy can be applied to obtain the selected base model training classes.

[0069] In one embodiment, the number of samples from different industries is unbalanced. The budget policy aims to balance the number of selected categories for each industry, while taking into account the differences in training difficulty, amount of available defect data, importance, etc., to balance the selected categories for different industries. The BIP method is used to select the best set of classes from the 1000 classes of ImageNet to train the basic model. A vector x is defined, where each binary element x_i represents class i to be included in the basic model training (x_i=1) or excluded (x_i=0). The BIP problem becomes:

[0070] maximize c T x

[0071] subject to Ax≤b

[0072] x={0,1} N

[0073] Where N is the total number of classes in the public dataset, which is equal to 1000 for ImageNet. c is a vector where each element c_i is defined as a coefficient that quantifies the overall benefit of selecting class i, which depends on the relevance of class i to the target task (p_(i, t)), the importance of each target task in the final application (r_t), and the balance factor of the accessible data size between different defect datasets (q_t). As shown below:

[0074]

[0075] Where t represents a specific target defect detection task, T is the total number of target tasks, and

[0076]

[0077] The smaller the data, the more difficult it is to train in transfer learning. When determining the selection of related categories in the public dataset, this category will be more favored (or compensated) so that closely related categories have higher weights to address the imbalance between different defect datasets.

[0078] This problem is subject to the budget policy constraint, that is, the total training budget of the basic model cannot exceed M, such as 50, 100. Therefore, the matrices A and b will satisfy:

[0079] a 1，j = l j =1

[0080] b 1 =M

[0081] In the formula, l_j is the cost of adding class j, and in the present invention, it is assumed that class j is uniformly 1. At this time, only one row of A and b is needed. However, more constraints can be added according to actual conditions.

[0082] Note that this BIP design works under the premise that training a base model with more classes can monotonically produce better predictions. For this premise, 50 classes should be safe in most cases because this model is a deep network that requires a lot of training data. However, when the number of classes reaches a certain value, the quality may increase or decrease. The range of the number of classes where the number of classes still has benefits can be determined empirically through experiments. In the specific case, the number of 50 classes is not too high, otherwise it will negatively affect the predictive ability of the final defect inspection model.

[0083] Finally, a cross-industry base model can be created by retraining the model with the new set of classes selected in the BIP process. This will be used for transfer learning to obtain the final defect model.

[0084] like Figure 5 Shown is an example of the common data categories selected for training a cross-industry model on the Wood and Texture dataset.

[0085] A specific example test of the method provided by the present invention is applied to verify the feasibility of the implementation of the method. Specifically: it includes two models for a single industry and a cross-industry basic model developed using the disclosed method, which is used to derive two special models. Figure 7a Shown is a texture dataset and Figure 7b The wood dataset shown.

[0086] For the development of cross-industry base models, a dataset is the DAGM-2007 dataset, which contains 8050 images for training, of which 1046 images contain defects; and 8050 images for testing, of which 1054 images contain defects. In this experiment, the training data is divided into two parts, 80% for training and 20% for validation during the training phase. Figure 7a Example images for each class included in the dataset are shown.

[0087] Another dataset used in this experiment is a wood defect dataset provided by previous researchers. Since the dataset cannot provide wood images without defects, some positive samples are also collected from the Internet. Sample images are as follows Figure 7b As shown, the wood data also uses an 80% training and 20% test data split.

[0088] This experiment is to retrain Inception V3 using a subset of ImageNet data, which is run on a computer with 8 graphics cards as servers. In order to achieve transfer learning of the defect detection classifier, it is run on a computer with graphics cards as clients.

[0089] A texture model was built on the basis of a single industry base model. Through the analysis of texture data, 500 classes were selected in ImageNet to train Inception V3. In the process of transfer learning, the weights of all layers are released for retraining, and the wood model is also built in a similar way.

[0090] For the cross-industry model, texture and wood data were used with the disclosed method described in this invention, and 50 classes were selected to retrain the Inception V3 model, and then transfer learning was applied to train a model for each wood and texture dataset.

[0091] The results of the above experiments are as follows: Compared with previous studies, the texture model based on a single industry base model shows higher accuracy.

[0092]

[0093] Table 1. Results of our defect detection model on texture data.

[0094]

[0095]

[0096] Table 2. Accuracy results of the texture data defect detection model.

[0097] The accuracy of the pre-trained base model method based on the wood dataset is 99.12%, while the accuracy of the built-in initial V3 is 97.7%. It can be seen that our framework using an embodiment for data augmentation shows high performance in defect detection on a limited dataset compared to existing methods.

[0098] Table 3 gives the test accuracy results and training time of the cross-industry base model and the texture and wood models built on it. Note that among the 50 classes selected from ImageNet, 40 classes rank high in wood, while the other 10 classes rank high in texture.

[0099] Model Accuracy Training time Basic Model 85.5％ 6.6hr Wood Model 100.0％ 6min Texture Model 99.7％ 54min

[0100] Table 3. Models involved in cross-industry methods (all trained on 8 GPUs).

[0101] From Table 3, we see that the basic models trained with relevance to two different industries can be used to obtain high-quality models for the defect inspection task for each industry. The time cost of each model is also satisfactorily low, which allows the server in our federated learning framework to efficiently update the model and prepare it for use by clients. The simulated FL process conducted in one cycle for 4 clients on the wood dataset can estimate the time cost and data size involved, as shown in Table 4.

[0102]

[0103] Table 4. Time cost and data size of the federated learning process involved in one cycle.

[0104] A new framework is disclosed that uses federated learning methods and deep learning models to provide manufacturers with automated defect detection services without sacrificing data privacy and requiring large datasets. In order to efficiently deploy and update the model, a cross-industry approach can be applied in the process. Experiments also prove the feasibility and effectiveness of cross-industry modeling under the FL framework. The framework provided by the present invention can protect the data privacy of customers by training models without directly accessing their original data. The framework allows both rapid deployment of models for new clients and continuous model updates that do not require large datasets to be generated for a single client. This method is expected to serve various industries in the manufacturing industry, continuously / efficiently improving model quality and rapidly deploying models.

[0105] The present invention also provides an electronic device, comprising: a processor;

[0106] A memory; and a program, wherein the program is stored in the memory and is configured to be executed by a processor, the program including instructions for executing a visual inspection method based on federated learning.

[0107] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement a visual detection method based on federated learning.

[0108] The present invention also provides a visual inspection system based on federated learning, such as Figure 8 As shown, it includes a request module 20, an update module 30, and a training module 10; wherein,

[0109] The request module 20 is configured to send a request instruction to the server to obtain the first basic model;

[0110] The updating module 30 is configured to collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model for product appearance inspection;

[0111] The training module 10 is configured to receive the updated server-side model sent by the server at a set threshold time and perform training to obtain a final model for performing the detection task; wherein the final model is a model that is updated once or multiple times from the first basic model.

[0112] Preferably, the updating module 30 includes a testing unit 310; the testing unit 310 is configured to test the defect image data using a pre-trained convolutional neural network model, and obtain the average probability of the defect image data being classified into each class in the public data set, so as to obtain the difference between each defect between the defect image data and the degree of association with each class in the public data set; and perform data expansion on the defect image data and generate an affinity matrix;

[0113] The affinity matrix is ​​normalized by the analytic hierarchy process to obtain the maximum of all defect classes;

[0114] The training module 10 includes a processing unit 110; the processing unit 110 is configured to perform model training using first defect image data of the industry and second defect image data in a public data set; wherein the first defect image data and the second defect image data are highly correlated; and

[0115] Performing data expansion and hierarchical analysis processing on the first defect image data and the second defect image data;

[0116] The normalized data of various industries in the processed public dataset are associated and appended together as an association matrix, and binary integer programming is used to obtain the final model.

[0117] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in any form. Any ordinary technician in the industry can smoothly implement the present invention as shown in the drawings and above. However, any equivalent changes, modifications and evolutions made by technicians familiar with the profession without departing from the scope of the technical solution of the present invention using the technical content disclosed above are all equivalent embodiments of the present invention. At the same time, any equivalent changes, modifications and evolutions made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the technical solution of the present invention.

Claims

1. A visual inspection method based on federated learning, characterized in that: The steps include: Sending a request instruction to the server to obtain a first basic model; Collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model; Receiving the updated server-side model sent by the server at a set threshold time and training it to obtain a final model for performing the detection task for product appearance detection; wherein the final model is a model updated once or multiple times by the first basic model; The method for obtaining the first basic model further includes: Refining the collected conventional data set according to the set specifications and performing convolutional neural network training to obtain the first basic model; The collecting defect image data to train the first basic model further includes: The defect image data is tested using a pre-trained convolutional neural network model, and the average probability of the defect image data being classified into each class in the public dataset is obtained to obtain the difference between each defect between the defect image data and the degree of association with each class in the public dataset; The step of receiving the updated server-side model sent by the server within the set threshold time and performing training further includes: The model is trained by using the first defect image data of the industry and the second defect image data in the public data set; wherein the first defect image data and the second defect image data are highly correlated.

2. The visual inspection method based on federated learning as claimed in claim 1, characterized in that: In the step of collecting defect image data to train the first basic model, it also includes: Perform data augmentation on defect image data and generate affinity matrix; The affinity matrix is ​​normalized by the analytic hierarchy process to obtain the maximum for all defect classes.

3. The visual inspection method based on federated learning as claimed in claim 1, characterized in that: In the simultaneous model training of the first defect image data and the second defect image data, the method further includes: Performing data expansion and hierarchical analysis processing on the first defect image data and the second defect image data; The normalized data of various industries in the processed public dataset are associated and appended together as an association matrix, and binary integer programming is used to obtain the final model.

4. An electronic device, characterized in that: include: processor; Memory; and a program, wherein the program is stored in the memory and configured to be executed by a processor, the program including instructions for executing the method according to claim 1.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is used by a processor to execute the method according to claim 1 .

6. A visual inspection system based on federated learning, characterized in that: It includes request module, update module and training module; among them, The request module is configured to send a request instruction to the server to obtain the first basic model; The updating module is configured to collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model; The training module is configured to send a request instruction to the server to obtain a first basic model; Refining the collected conventional data set according to the set specifications and performing convolutional neural network training to obtain the first basic model; Collect defect image data to train the first basic model and extract high-level features therein and send them to the server to update the server-side model; Receiving the updated server-side model sent by the server at a set threshold time and training it to obtain a final model for performing the detection task for product appearance detection; wherein the final model is a model updated once or multiple times by the first basic model; The updating module includes a testing unit; the testing unit is configured to test the defect image data using a pre-trained convolutional neural network model, and obtain the average probability of the defect image data being classified into each class in the public data set, so as to obtain the difference between each defect between the defect image data and the degree of association with each class in the public data set; The training module includes a processing unit; the processing unit is configured to use the first defect image data of this industry and the second defect image data in the public data set to jointly perform model training; wherein the first defect image data and the second defect image data are highly correlated.

7. The visual inspection system based on federated learning as claimed in claim 6, characterized in that: The testing unit is further configured to perform data expansion on the defect image data and generate an affinity matrix; The affinity matrix is ​​normalized by the analytic hierarchy process to obtain the maximum of all defect classes; The processing unit is further configured to perform data expansion and hierarchical analysis processing on the first defect image data and the second defect image data; The normalized data of various industries in the processed public dataset are associated and appended together as an association matrix, and binary integer programming is used to obtain the final model.

Images