A low-rank federated generalization method, device and equipment for microscopic image defect recognition

Abstract

The present application relates to a kind of hierarchical domain invariant low rank federal generalization method, device and equipment of microscopic image defect identification, method includes: step S1, in server end initialization defect classification model, obtain respective microscopic image data in multiple clients, low rank adaptation module is set in defect classification model;Step S2, defect classification model is issued to each client, each client is trained to the low rank adaptation module of defect classification model by respective microscopic image data, after training, low rank adaptation module is uploaded to server end;Step S3, the update direction of the low rank adaptation module of training completion is calculated;Step S4, according to update direction, aggregate weight is calculated;Step S5, according to the low rank adaptation module of training completion and its aggregate weight, new defect classification model is obtained.The present application reduces communication overhead while improving the generalization ability of microscopic image defect classification model under unknown imaging domain.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a low-rank federated generalization method, apparatus and device for microscopic image defect recognition. Background Technology

[0002] With the development of microscopic optical imaging, industrial cameras, automated inspection equipment, and deep learning technology, image recognition-based industrial defect detection methods have been gradually applied to industrial inspection scenarios such as semiconductors, new energy, and precision manufacturing. In practical applications, microscopic images can be used for defect identification and quality inspection of objects such as wafers, chip packages, battery electrodes, welded areas, precision metal parts, glass covers, and thin film materials.

[0003] In microscopic image defect recognition tasks, images acquired from different customers, factories, equipment, or production lines often exhibit significant distributional differences. These differences may stem from factors such as microscopic equipment, industrial cameras, light source conditions, magnification, exposure parameters, material batches, and acquisition environments. Even when the object being detected and the defect category are the same, images from different sources may still differ in brightness, contrast, texture detail, noise level, background appearance, and localized reflections. If the defect classification model is trained only under single or limited imaging conditions, it is prone to problems such as decreased recognition accuracy, increased false positive rate, or increased false negative rate when encountering unknown equipment, unknown batches, or unknown acquisition environments during the testing phase.

[0004] Federated learning enables multiple clients to collaboratively build models through local training and server aggregation without sharing raw image data, making it suitable for industrial detection scenarios with high data privacy requirements. However, traditional federated learning typically requires frequent transmission of complete model parameters between clients and servers. As deep learning models continue to grow in size, full-parameter federated training incurs high communication overhead and computational burden, making deployment difficult in edge devices, industrial control computers, mobile terminals, or bandwidth-constrained industrial environments. To reduce communication overhead, low-rank adaptation techniques are used for parameter-efficient model training. This technique freezes the basic model parameters and trains only a small number of low-rank matrix parameters, significantly reducing the size of trainable and communication parameters. In federated learning scenarios, clients only upload low-rank parameters, not the complete model parameters, effectively reducing communication costs.

[0005] However, directly applying low-rank adaptation techniques to federated domain generalization tasks still has shortcomings. First, the expressive power of the low-rank parameter space is limited; simple averaging aggregation may not adequately characterize the domain-invariant features shared among different clients. Second, the low-rank matrix itself is not equivalent to the parameter increments used during final inference; directly measuring the consistency of updates across different clients in the low-rank matrix space may not accurately reflect the model's expression direction in the full-rank parameter space. Third, different network layers differ in model transferability and domain sensitivity; uniformly applying the same aggregation to all layers may easily overlook the different contributions of layer features to cross-domain generalization ability. Summary of the Invention

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0007] This invention provides a hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification, comprising the following steps:

[0008] Step S1: Initialize the defect classification model on the server side and acquire the microscopic image data of each client. The defect classification model is equipped with a low-rank adaptation module.

[0009] Step S2: The defect classification model is distributed to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image data. The basic model parameters in the defect classification model are kept frozen. After training is completed, the low-rank adaptation module is uploaded to the server.

[0010] Step S3: Calculate the update direction of the trained low-rank adaptation module;

[0011] Step S4: Calculate the aggregation weight based on the update direction;

[0012] Step S5: Based on the trained low-rank adaptation module and aggregate weights, a new defect classification model is derived.

[0013] Preferably, the defect classification model adopts the ViT-B / 32 visual Transformer model, which includes an image patch embedding layer, classification label, position encoding, Transformer encoder and defect classification layer. The Transformer encoder includes a multi-head self-attention module and a feedforward network module. The low-rank adaptation module is only embedded into the linear mapping layer corresponding to the query projection matrix and value projection matrix in the multi-head self-attention module.

[0014] Preferably, in step S3, calculating the update direction of the trained low-rank adaptation module includes:

[0015]

[0016]

[0017] in, This represents the full-rank weight increment of the nth-layer low-rank adaptation module on the server side. and This represents the low-rank matrix of the nth-level low-rank adaptation module on the server side. This represents the full-rank weight increment of the low-rank adaptation module at layer n for the k-th client. This indicates the update direction of the low-rank adaptation module at layer n for the k-th client.

[0018] Preferably, in step S4, calculating the aggregation weight based on the update direction includes:

[0019] Calculate the domain invariance score based on the update direction;

[0020] The aggregation weight is calculated based on the domain invariance score.

[0021] Preferably, the domain invariance score is calculated based on the update direction:

[0022]

[0023]

[0024] in, Indicates the first The client and the first The similarity of the update directions of each client at layer n Represents matrix vectorization. Indicates the inner product. Indicates the modulus length;

[0025] The domain invariance score is represented by K, where K represents the number of clients.

[0026] The aggregate weights are calculated based on the domain invariance score:

[0027]

[0028] This represents the aggregate weight of the k-th client at layer n.

[0029] Preferably, in step S5, deriving a new defect classification model based on the trained low-rank adaptation module and its aggregate weights includes:

[0030]

[0031]

[0032]

[0033] This represents the weight matrix of the nth base layer. This represents the update weight matrix of the nth layer of the model.

[0034] Preferably, in step S1, different clients correspond to different customers, different factories, different production lines, different microscopes, different light source conditions, different microscope magnifications, different product batches, or different image domains in a publicly available multi-domain dataset.

[0035] Preferably, in step S2, the microscopic image data of each client is kept locally and is not uploaded to the server or shared between clients.

[0036] The present invention also provides a hierarchical domain-invariant low-rank federated generalization device for microscopic image defect recognition, comprising:

[0037] The initialization module is used to initialize the defect classification model on the server side and acquire microscopic image data from multiple clients. The defect classification model includes a base layer and a low-rank adaptation module.

[0038] The training module is used to distribute the defect classification model to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image data and uploads the trained low-rank adaptation module to the server.

[0039] The update module is used to calculate the update direction of the trained low-rank adaptation module;

[0040] The weighting module is used to calculate the aggregate weight based on the update direction;

[0041] The aggregation module is used to derive a new defect classification model based on the trained low-rank adaptation module and its aggregation weights.

[0042] The present invention also provides an electronic device, including a memory and a processor, characterized in that the memory stores a computer program, and the processor is configured to run the computer program to perform the above-described hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification.

[0043] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the above-described hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification.

[0044] The present invention has at least the following beneficial effects:

[0045] This invention addresses the problems of the inability to centrally share industrial microscopic image data from multiple clients, differences in imaging distribution among different clients, high communication overhead in full-parameter federated training, and difficulty in guaranteeing cross-domain generalization ability through low-rank parameter aggregation. By introducing a low-rank adaptation module into the defect classification model and constructing a layer-by-layer domain invariance evaluation and aggregation mechanism based on the full-rank weight increment after reparameterization, the invention improves the generalization ability of the microscopic image defect classification model in unknown imaging domains while reducing communication overhead. Attached Figure Description

[0046] Figure 1 This is a flowchart of the hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to an embodiment of the present invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0049] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0050] like Figure 1 As shown, embodiments of the present invention provide a hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification, comprising the following steps:

[0051] Step S1: Initialize the defect classification model on the server side and acquire the microscopic image data of each client. The defect classification model is equipped with a low-rank adaptation module.

[0052] Build by A federated training system consisting of [number] clients, denoted as: [client dataset].

[0053]

[0054] in, Indicates the first Microscopic image data from individual clients. Different clients may correspond to different customers, factories, production lines, microscopic equipment, light source conditions, magnifications, product batches, or different image domains in publicly available multi-domain datasets. The microscopic image data from each client can be represented as:

[0055]

[0056] in, Indicates the first The first client Zhang's microscopic images, This indicates the corresponding category label, defect label, or segmentation label. Indicates the first Number of samples in each client.

[0057] The server initializes the defect classification model and inserts low-rank adaptor modules into one or more trainable network layers in the model. Let the... The base weights of the network layer that inserts low-rank adaptation modules are:

[0058]

[0059] The low-rank adaptation module consists of two low-rank matrices, namely:

[0060]

[0061]

[0062] in, It is a low-rank dimension and satisfies:

[0063]

[0064] The defect classification model can adopt the ViT-B / 32 visual Transformer model, which includes an image patch embedding layer, classification label, position encoding, Transformer encoder and defect classification layer. The Transformer encoder includes a multi-head self-attention module and a feedforward network module. The low-rank adaptation module is only embedded into the linear mapping layer corresponding to the query projection matrix and the value projection matrix in the multi-head self-attention module.

[0065] Step S2: The defect classification model is distributed to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image data. The basic model parameters in the defect classification model are kept frozen. The trained low-rank adaptation module is then uploaded to the server.

[0066] In each round of federated communication, the server sends the current defect classification model parameters to each client. The microscopic image data of each client is stored locally and is neither uploaded to the server nor shared between clients.

[0067] After receiving the data, the client performs training on local microscopic image data. During training, the parameters of the base layer... Keep frozen, only low-rank matrices and Participate in gradient updates. After completing local training, the client uploads the updated low-rank parameters of each layer to the server:

[0068]

[0069] in, This indicates the number of network layers in the defect classification model where low-rank adapter modules are inserted. Since the uploaded parameters are only low-rank matrices, rather than the complete model parameters, the communication overhead between the client and server can be reduced.

[0070] Step S3: Calculate the update direction of the trained low-rank adaptation module.

[0071] The day after training The first client The full-rank weight increment of the low-rank adaptation module is:

[0072]

[0073] The server-side not trained The full-rank weight increment of the low-rank adaptation module is:

[0074]

[0075] Then the first The first client The update direction of the low-rank adaptation module is as follows:

[0076]

[0077] Right now:

[0078]

[0079] The full-rank weight increment is used to characterize the actual update direction of the client relative to the server.

[0080] Step S4: Calculate the aggregation weight based on the update direction.

[0081] For the A low-rank adaptation module is used by the server to calculate the directional similarity of the full-rank weight increments between different clients. The client and the first The client in the first The similarity of the update direction of a layer is represented as:

[0082]

[0083] in, Indicates the first The client and the first The similarity of the update directions of each client at layer n Represents matrix vectorization. Indicates the inner product. Indicates the modulus length;

[0084] Based on this, calculate the first... The client in the first Layer domain invariance score:

[0085]

[0086] K represents the number of clients. When When it is larger, it indicates the first The client in the first The update direction of the layer is more consistent with other clients, and it is more likely to contain domain-invariant information shared across clients; when A smaller value indicates that the update direction may contain more client-specific information.

[0087] To use the domain invariance score for parameter aggregation, the server performs a certain number of tests on each client in the first round. The scores of each layer are normalized to obtain the layer-by-layer aggregated weights:

[0088]

[0089] in, Indicates the first The client in the first The layer's aggregate weights. These aggregate weights are used to increase the contribution of low-rank parameters whose update direction is consistent with that of most clients, and to reduce the impact of domain-specific updates on the global model.

[0090] Step S5: Based on the trained low-rank adaptation module and its aggregate weights, a new defect classification model is derived.

[0091] The server, according to the first The aggregation weights of the layers are used to perform weighted aggregation of the low-rank matrices uploaded by each client:

[0092]

[0093]

[0094] and These represent the aggregated low-rank matrices. This process is executed independently in each low-rank adaptation module, thus forming a hierarchical, fine-grained federated aggregation mechanism.

[0095] The server constructs the next round of the model based on the aggregated low-rank parameters. The updated weights of the layers are represented as follows:

[0096]

[0097] The server will distribute the updated global low-rank parameters to each client to begin the next round of federated training until the model training is complete.

[0098] While maintaining the advantages of low-rank adaptive communication, this method can evaluate the consistency of update directions of different clients from the full-rank weight increment space after reparameterization, and perform fine-grained aggregation based on the domain invariance of different network layers, thereby improving the generalization ability of the global defect classification model in the unknown microscopic imaging domain.

[0099] Embodiments of the present invention also provide a hierarchical domain-invariant low-rank federated generalization device for microscopic image defect recognition, comprising:

[0100] The initialization module is used to initialize the defect classification model on the server side and obtain microscopic image training data from multiple clients. The defect classification model includes a base layer and a low-rank adaptation module.

[0101] The training module is used to distribute the defect classification model to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image training data, and then uploads the trained low-rank adaptation module to the server.

[0102] The update module is used to calculate the update direction of the trained low-rank adaptation module;

[0103] The weighting module is used to calculate the aggregate weight based on the update direction;

[0104] The aggregation module is used to derive a new defect classification model based on the trained low-rank adaptation module and its aggregation weights.

[0105] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the corresponding method embodiments belong to the same concept, and the specific implementation process can be found in the corresponding method embodiments, which will not be repeated here.

[0106] Furthermore, according to an embodiment of the present invention, an electronic device is also provided, the electronic device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the method provided in the above embodiments.

[0107] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0108] The method embodiments provided in this invention can be executed in a terminal, server, or similar computing device; that is, the aforementioned electronic device may include a terminal, server, or similar computing device. Servers can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) (e.g., one or more processors) and memory, and one or more storage media (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media may be temporary or persistent storage. The program stored in the storage media may include one or more modules, each module including a series of instruction operations on the server. Furthermore, the CPU may be configured to communicate with the storage media and execute a series of instruction operations stored in the storage media on the server. The server may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input / output interfaces, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0109] Input / output interfaces can be used to receive or send data via a network. Specific examples of such networks may include a wireless network provided by the server's communication provider. In one example, the input / output interface includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface may be a Radio Frequency (RF) module used for wireless communication with the Internet. Wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0110] One embodiment of the present invention also provides a computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program for implementing the above-described method, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method in the above-described embodiment.

[0111] Optionally, in embodiments of the present invention, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0112] One embodiment of the present invention also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method provided in the above embodiments.

[0113] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0114] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0115] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0116] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A hierarchical domain-invariant low-rank federated generalization method for defect identification in microscopic images, characterized in that, Includes the following steps: Step S1: Initialize the defect classification model on the server side and acquire the microscopic image data of each client. The defect classification model is equipped with a low-rank adaptation module. Step S2: The defect classification model is distributed to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image data. The basic model parameters in the defect classification model are kept frozen. After training is completed, the low-rank adaptation module is uploaded to the server. Step S3: Calculate the update direction of the trained low-rank adaptation module; Step S4: Calculate the aggregation weight based on the update direction; Step S5: Based on the trained low-rank adaptation module and aggregate weights, a new defect classification model is derived.

2. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 1, characterized in that, The defect classification model adopts the ViT-B / 32 visual Transformer model, which includes an image patch embedding layer, classification label, position encoding, Transformer encoder and defect classification layer. The Transformer encoder includes a multi-head self-attention module and a feedforward network module. The low-rank adaptation module is only embedded into the linear mapping layer corresponding to the query projection matrix and value projection matrix in the multi-head self-attention module.

3. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 2, characterized in that, In step S3, the update direction of the trained low-rank adaptation module is calculated, including: in, This represents the full-rank weight increment of the nth-layer low-rank adaptation module on the server side. and This represents the low-rank matrix of the nth low-rank adaptation module on the server side. This represents the full-rank weight increment of the low-rank adaptation module at layer n for the k-th client. This indicates the update direction of the low-rank adaptation module at layer n for the k-th client.

4. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 3, characterized in that, In step S4, the aggregation weights are calculated based on the update direction, including: Calculate the domain invariance score based on the update direction; The aggregation weight is calculated based on the domain invariance score.

5. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 4, characterized in that, The domain invariance score is calculated based on the update direction: in, Indicates the first The client and the first The similarity of the update directions of each client at layer n Represents matrix vectorization. Indicates the inner product. Indicates the modulus length; The domain invariance score is represented by K, where K represents the number of clients. The aggregate weights are calculated based on the domain invariance score: This represents the aggregate weight of the k-th client at layer n.

6. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 5, characterized in that, In step S5, a new defect classification model is derived based on the trained low-rank adaptation module and its aggregate weights, including: This represents the weight matrix of the nth base layer. This represents the update weight matrix of the nth layer of the model.

7. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 1, characterized in that, In step S1, different clients correspond to different customers, different factories, different production lines, different microscope equipment, different light source conditions, different microscope magnification, different product batches, or different image domains in a public multi-domain dataset.

8. The hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification according to claim 1, characterized in that, In step S2, the microscopic image data of each client is kept locally and is not uploaded to the server or shared between clients.

9. A hierarchical domain-invariant low-rank federated generalization device for microscopic image defect recognition, characterized in that, include: The initialization module is used to initialize the defect classification model on the server side and acquire microscopic image data from multiple clients. The defect classification model includes a base layer and a low-rank adaptation module. The training module is used to distribute the defect classification model to each client. Each client trains the low-rank adaptation module of the defect classification model using its own microscopic image data and uploads the trained low-rank adaptation module to the server. The update module is used to calculate the update direction of the trained low-rank adaptation module; The weighting module is used to calculate the aggregate weight based on the update direction; The aggregation module is used to derive a new defect classification model based on the trained low-rank adaptation module and its aggregation weights.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the hierarchical domain-invariant low-rank federated generalization method for microscopic image defect identification as described in any one of claims 1-8.

Images