Product package quality inspection classification method based on federal transfer learning

Abstract

The invention discloses a product package quality inspection classification method based on federal transfer learning. The method comprises the steps that a training client locally uses data resampling and a multi-scale feature pyramid network to process product package data; the training client carries out input training on the data until the local model converges; second-order statistical difference weighted aggregation is used to obtain a global model, a bidirectional backflow migration method is used to optimize the model, and an optimized model is obtained; and the target client receives the global model sent by the server for quality inspection and classification of the packaging data of the to-be-tested product. According to the method, the quality inspection classification capability is learned and applied to the target domain user under the scene of facing multi-source domain and privacy-protecting product package quality inspection classification, the model is trained in cooperation with multiple participants, and the generalization capability and accuracy of the model are improved under the condition that the data volume is limited and the computing resources are limited.

Description

Technical Field

[0001] The present invention relates to the technical field of federated learning and product packaging classification quality inspection, and in particular to a product packaging quality inspection classification method based on federated transfer learning. Background Art

[0002] In modern industrial production, packaging, as an important component of products, not only plays a role in protecting products, but also directly affects the market sales performance of products. High-quality packaging can effectively prevent products from being damaged during storage, transportation and sales, while increasing consumers' desire to buy. However, if the packaging does not meet relevant specifications or has defects, it will lead to a decline in consumer experience at best, or product quality problems at worst, and may even trigger large-scale product recalls, causing huge economic losses and credibility crises to companies. Therefore, packaging testing and quality control have become a crucial link in industrial production.

[0003] For a long time, packaging quality inspection has mainly relied on manual operation. Manual quality inspection is mainly carried out by quality inspectors through naked eye observation, manual testing, etc. Although this method can meet the needs of some simple inspections, it is highly dependent on manpower, not only the inspection cost is high, but also there are many shortcomings that are difficult to overcome. First, the accuracy and efficiency of human visual observation are often affected by the physical state of the quality inspector, such as fatigue, distraction or lack of experience, which may lead to missed inspections or misjudgments. Secondly, in the face of high-intensity and repetitive inspection tasks, manual quality inspection is difficult to maintain long-term stability and consistency. These drawbacks make it difficult for traditional packaging quality inspection methods to meet the large-scale, high-efficiency, and high-quality production needs of modern industry.

[0004] With the continuous advancement of science and technology, especially the rapid development of artificial intelligence and machine learning technology, automated packaging quality inspection technology based on visual systems has emerged and quickly gained widespread attention and application. This type of technology uses computer vision methods to perform non-contact inspection of packaging, and uses high-precision cameras, sensors and algorithm models to automatically detect surface defects, structural integrity and packaging information of packaging. Compared with traditional manual inspection, this technology has shown great advantages in many aspects. First, automated quality inspection can significantly increase the speed of inspection, especially for high-capacity assembly line production. Second, by introducing deep learning algorithms, automated quality inspection systems can identify subtle packaging defects, thereby greatly improving inspection accuracy. In addition, this type of technology can also support long-term operation without being affected by human factors such as fatigue, effectively reducing the cost of manual inspection and the rate of false positives. Therefore, automated quality inspection technology based on visual systems has become an indispensable and important means in modern industry, providing a strong guarantee for improving production efficiency and product quality.

[0005] However, most current vision system packaging inspection solutions still face some practical problems and challenges. A prominent limitation is that these methods usually rely on a large amount of labeled data to improve detection accuracy. The training of deep learning models requires the use of a large number of high-quality, well-labeled data samples, especially those involving packaging defects. However, in actual industrial environments, it is not easy to obtain a large amount of labeled data. On the one hand, defect data may be scarce in some production lines, and companies need to spend a lot of time manually collecting defect samples from normal production. On the other hand, the labeling process usually requires professionals to operate, which is not only costly but also takes up a lot of human resources. In addition, even if sufficient training data is obtained, traditional deep learning methods perform poorly in dealing with differences between different data domains. For example, changes in production line equipment, different packaging materials, fluctuations in lighting conditions, and other environmental factors may cause significant changes in data distribution, which greatly reduces the detection effect of the model in the new environment.

[0006] In order to solve the above problems, federated transfer learning technology has gradually been introduced into the field of industrial defect detection. Federated transfer learning collaborates with multiple participants to jointly train models without sharing original data, and then draws on the knowledge of existing models to improve the model performance on the target task. However, in the case of multiple source domains and unknown categories, how to effectively combine federated transfer learning technology to achieve cross-domain knowledge transfer and model optimization is still a difficult problem that needs to be solved. Summary of the invention

[0007] The purpose of the present invention is to provide a product packaging quality inspection and classification method based on federated transfer learning. In the face of multi-source domains and imbalanced category data, the knowledge of training clients in different source domains is migrated to target clients in the target domain. Without sharing the original data, multiple participants are coordinated to jointly train models and thresholds, so as to improve the generalization ability and accuracy of the model under the condition of limited data volume and limited computing resources.

[0008] To achieve the above object, the present invention provides a product packaging quality inspection classification method based on federated transfer learning, comprising the following steps: S1. Data preparation Collect packaging data from the local product production process, train the client to use data resampling and multi-scale feature pyramid network to process the data, and obtain multi-scale features of product packaging data.

[0009] S2. Local model training The training client inputs the processed data in S1 for training. The training process uses a modified loss function that dynamically superimposes the cross entropy loss function and the domain alignment loss function until the local model converges.

[0010] S3, model update and upload The training client uploads the trained model parameters and local data feature dataset to the server.

[0011] S4. Model weight calculation and aggregation The server receives the model parameters, local data features, and target client data features uploaded by the training client, and uses the second-order statistical difference weighted aggregation and two-way reflux migration method for aggregation and migration. The second-order statistical difference weighted aggregation method measures the difference between the local data features and the target client data features, and performs weighted aggregation on the model parameters of the training client to obtain a global model. The two-way reflux migration method uses the target client data features for model migration to obtain an optimized global model.

[0012] S5. Model deployment and inference The target client receives the global model and global threshold sent by the server, which are used to classify the data to be tested for packaging quality inspection.

[0013] Preferably, in S1, the training client uses a data resampling method to balance data categories on the collected product packaging data, and uses a multi-scale feature pyramid network to extract multi-scale features of the data set.

[0014] Preferably, the multi-scale feature comprises the following steps: S101, balance the data using a data resampling method, oversample a small number of category samples, and undersample a majority of category samples to obtain a balanced data set; S102. Use a multi-scale feature pyramid network to extract features from the balanced data set, extract different features through feature networks of different sizes, and fuse them to obtain multi-scale features.

[0015] Preferably, in the second-order statistical difference weighted aggregation method, the model weight calculation is specifically to measure the CORAL distance between the local data feature data set and the second data feature data set, and calculate the model weight of a single training client.

[0016] Therefore, the present invention adopts the above-mentioned product packaging quality inspection and classification method based on federated transfer learning. When faced with the problem scenario of multi-source domains and unknown category data classification in the test data, the classification ability of unknown categories is learned in multiple source domains and applied to the target domain users. Without sharing the original data, multiple participants are coordinated to jointly train the model and threshold, so as to improve the generalization ability and accuracy of the model under the condition of limited data amount and limited computing resources.

[0017] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is an overall framework diagram of an embodiment of a product packaging quality inspection and classification method based on federated transfer learning of the present invention; Figure 2 It is a flowchart of the steps of a product packaging quality inspection and classification method according to an embodiment of a product packaging quality inspection and classification method based on federated transfer learning of the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0020] Unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0021] Embodiment 1 The present invention provides a product packaging quality inspection and classification method based on federated transfer learning, and its overall framework is shown in the following figure: Figure 1 As shown, it includes a training client, a target client and a server, wherein the training client and the server are connected for communication, and the target client transmits data features to the server. Figure 2 This is a flowchart of the steps of the product packaging quality inspection classification method. In this embodiment, the federated transfer learning process is defined under the problem scenarios of multi-source domains and data privacy protection, including the data used, the data category and data quantity set by each user, the different data sets for the training client and the target client, and the hyperparameters of the model. The specific process includes the following steps: S1. Data preparation and preprocessing Each training client device collects product packaging data during the production process locally, including image data, and performs preprocessing operations such as standardization and normalization. Using the data resampling scheme, the minority category samples are oversampled and the majority category samples are undersampled to obtain a balanced data set.

[0022] A multi-scale feature pyramid network is used to extract data features, and the features extracted by feature extraction networks of different sizes are fused to obtain the feature vectors of the packaging appearance data collected by each training client.

[0023] S2. Local model training The training client inputs the feature vector processed in step S1 for training and selects the DNN model for training: Define the dynamic superposition of the cross entropy loss function and the domain alignment loss function, the loss function L total The specific calculation formula is as follows: ; Among them, L CE Expressed as the cross entropy loss term, Indicates the predicted category, y c represents the true category, L AL Denoted as the domain alignment loss term, f s (x i ) is represented as the feature representation of the source domain sample, f t (x i ’ ) is represented as the feature representation of the target domain sample, M represents the number of categories, c represents the category number, x i is the source domain sample, x i ’ represents the target domain sample, N represents the number of samples, and θ represents L AL The weight of .

[0024] The optimizer updates the model parameters. The specific formula is as follows: ; in, W are model parameters, b is the deviation term, η is the learning rate until the local model converges and obtains the local model.

[0025] S3, model update and upload Each training client device uploads the locally trained model parameters and local data features to the server. The uploaded parameters are model parameter information, and the data features include feature vectors of source user data.

[0026] S4. Model weight calculation and model update The server receives the model parameters and data features uploaded by each training client, as well as the data features uploaded by the target client. Based on the data features of the training client and the target client, a two-layer optimization strategy is used to optimize the model and calculate the CORAL distance.

[0027] For U source domain training clients, the model parameters of each client are input into the central server, and the federated migration process is carried out by using the second-order statistical difference weighted aggregation and bidirectional reflux migration method. The second-order statistical difference weighted aggregation is to update the global model parameters ω global , to minimize the weighted sum of the loss functions of all clients. This can be achieved through global model aggregation, where the local model parameters ω of each client are u,local are aggregated to update the global model ω global . Use the formula: ; where ω t global is the global model parameter of the tth round of communication, F u (ω t global ) is the loss function of the source domain training client u, and U is the total number of source domain training clients.

[0028] Using the global model aggregation process, weights are made based on the CORAL distance between the data features in each source domain training client and the data features in the target domain client. First, the covariance matrix C of the source domain training client u and the target domain client data is calculated. u,s and C T , using the formula: ; Among them, n S and n T are the number of samples in the source domain and the target domain, respectively, u S is the dataset of source domain training client u, D T is the target client dataset.

[0029] According to the obtained covariance matrix, the CORAL distance between the source domain training client u and the target domain is calculated using the formula: ; in represents the Frobenius norm of the matrix, and d is the dimension of the data features.

[0030] Among U source domain training clients, source domain training client u provides data D u S , upload the local model, and calculate the weight corresponding to the model according to the obtained distance. The specific expression for model weight calculation is: ; Among them, α u is the weight of the source domain training client u, D u,coralIt measures the CORAL distance between the source domain training client u and the target client dataset. The weight of each model is obtained according to the different CORAL distance calculations.

[0031] The specific expression of global aggregation is as follows: ; Among them, α u is the weight of model u, ω u global is the model of client u trained in the source domain.

[0032] According to the weight of each model parameter obtained above, the server aggregates the models of all training clients to obtain the global model. Using the weighted aggregation method, the model is integrated according to the weight of each training client.

[0033] The obtained global model is subjected to a two-way backflow migration strategy based on the data characteristics of the target domain, and the obtained global model is migrated and optimized. Specifically, the data characteristics of the target domain are used to optimize the model parameters of the target client. This is achieved by training the model on the local data of the target client. The formula used is:

[0034] where ω target t+1 is the model parameter of the target client after t rounds of communication, F(ω target t+1 ;F T ) is the loss function of the target client, F T It is the data feature of the target client, and the optimized global model is deployed on the target client for classification.

[0035] S5. Model deployment and inference The target client receives the optimized global model sent by the server and applies it to the data to be tested for product packaging quality inspection and classification. The target client uses the updated global model to perform product packaging quality inspection and classification on the data in the actual production environment to obtain the classification results.

[0036] Therefore, the present invention adopts the above-mentioned product packaging quality inspection and classification method based on federated transfer learning. In the face of multi-source domains and data privacy protection problem scenarios, the product packaging quality inspection and classification capabilities are learned in multiple source domains, and applied to target domain users. Without sharing the original data, the model is trained in collaboration with multiple participants to improve the generalization ability and accuracy of the model under the conditions of limited data volume and limited computing resources.

[0037] Based on the same technical solution, the present invention also provides an electronic device, including: Memory for storing computer programs; A processor is used to implement the steps of the above-mentioned product packaging quality inspection and classification method based on federated transfer learning when executing the computer program.

[0038] Based on the same technical solution, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the product packaging quality inspection and classification method based on federated transfer learning are implemented. The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0039] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0040] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A product packaging quality inspection and classification method based on federated transfer learning, characterized by: The following steps are involved: S1. Each training client collects packaging appearance data generated in the local production process and performs data balancing, and performs multi-scale feature extraction on the balancing result to obtain a feature vector of the packaging appearance data collected by each training client; S2, each training client uses the feature vector of the packaging appearance data collected by each training client obtained in S1 as a source domain sample to train the local model, and uploads the trained local model of each training client and the data features of each training client to the server; wherein the data features of each training client are the feature vectors of the packaging appearance data collected by each training client; S3, the target client collects the packaging appearance data generated in the local production process, and uploads the feature vector extracted by the feature extractor as the target client data feature as the target domain sample to the server; S4, the server receives the trained local models of each training client and the data features of each training client and the data features of the target client, aggregates and migrates the trained local models of each training client to obtain an optimized global model, and sends it to the target client; S5. The target client uses the received global model to complete product packaging quality inspection and classification.

2. According to claim 1, a product packaging quality inspection and classification method based on federated transfer learning is characterized by: In S1, the training client uses a multi-scale feature pyramid network to extract multi-scale features to obtain multi-scale features.

3. According to claim 1, a product packaging quality inspection and classification method based on federated transfer learning is characterized by: Select the DNN model as the local model, and its loss function L total Defined as the cross entropy loss term L CE and domain alignment loss L AL Dynamic overlay: ； in, Indicates the predicted category, y c represents the true category, M represents the number of categories, c represents the category number, x i is the source domain sample, f s (x i ) is represented by x i The feature representation of x i ’ represents the target domain sample, f t (x i ’ ) is represented by x i ’ The feature representation, N represents the number of samples, θ represents L AL The weight of .

4. According to claim 1, a product packaging quality inspection and classification method based on federated transfer learning is characterized by: In S4, the global model is obtained by weighted aggregation using second-order statistical differences, including: Based on the associated alignment CORAL distance between the data features of each training client and the data features of the target client, the weight of the local model of each training client is calculated, and the global model is obtained by weighted aggregation of the local models of each training client.

5. According to claim 4, a product packaging quality inspection and classification method based on federated transfer learning is characterized by: The weight α of the u-th training client local model u The calculation formula is: ； Among them, D u,coral is the CORAL distance between the data features of the u-th training client and the data features of the target client, and U is the number of training clients.

6. The method for product packaging quality inspection and classification based on federated transfer learning according to claim 5 is characterized by: The calculation formula of the CORAL distance between the u-th training client data feature and the target client data feature is: ； in, represents the Frobenius norm of the matrix, d is the dimension of the data feature, C u,s and C T are the covariance matrices of the packaging appearance data collected by the u-th training client and the packaging appearance data collected by the target client, respectively.

7. The method for product packaging quality inspection and classification based on federated transfer learning according to claim 1, characterized in that: In S4, the bidirectional reflux migration method is used to perform model migration optimization to obtain an optimized global model.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

9. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the method according to any one of claims 1 to 7 when executing the computer program.

Images