Industrial anomaly detection method and device based on cloud edge collaboration

Abstract

The invention discloses an industrial anomaly detection method and device based on cloud-edge cooperation. The method comprises the steps that an industrial site configures a cloud server comprising a large model and an edge client comprising an anomaly detection model; a camera is used for collecting image data of products on the industrial production line in real time and marking the image data; using the image data to train an anomaly detection model at an edge client; the trained anomaly detection model parameters are uploaded to a cloud server for aggregation, an integrated soft label is generated through cooperation of the large model and the multiple small models, and the integrated soft label is used for performing parameter updating on the anomaly detection model of the edge client; training and updating of the anomaly detection model are cycled, and real-time collected image data are detected in parallel. According to the method, the generalization ability of the large model and the local experience of the small model are fused, the collective intelligence of the model is enhanced, the accuracy, the stability and the cross-production-line generalization ability of anomaly detection on the industrial assembly line are remarkably improved, and the real-time requirement under the condition that edge device resources are limited is met.

Description

Technical Field

[0001] The present invention relates to the field of industrial Internet of Things, and particularly to an industrial anomaly detection method and device based on cloud-edge collaboration. Background Art

[0002] In the field of industrial Internet of Things, product anomaly detection is an important link to ensure the quality of products on the production line. Due to their superior performance in image processing and pattern recognition, deep learning and neural network technologies are widely used in industrial product anomaly detection tasks, such as key fields like surface defect detection and product defect identification. However, product anomaly detection still faces some technical challenges. Traditional centralized methods rely on cloud servers to process and analyze the image data collected from the production line. Although they can provide high detection accuracy, due to limitations in network bandwidth, transmission delay, and data privacy issues, it is difficult to meet the requirements of industrial scenarios for detection real-time performance and data security. In addition, existing technologies usually focus on improving detection accuracy, resulting in an overly large model size, making it difficult to operate efficiently on edge devices with limited computing resources. How to achieve efficient product anomaly detection in a distributed environment while taking into account real-time performance and resource utilization is a technical problem that urgently needs to be solved currently. Summary of the Invention

[0003] In view of the above problems, the present invention proposes an industrial anomaly detection method and device based on cloud-edge collaboration to achieve efficient and real-time product anomaly detection in the industrial field. To achieve the above invention objective, the present invention proposes the following technical solutions: An industrial anomaly detection method based on cloud-edge collaboration, comprising the following steps:

[0004] S1. Configure a cloud server containing a large model and an edge client containing an anomaly detection model at the industrial site;

[0005] S2. Use a camera to collect and annotate the image data of products on the industrial production line in real time;

[0006] S3. Use the image data to train the anomaly detection model on the edge client; upload the parameters of the trained anomaly detection model to the cloud server for aggregation, generate integrated soft labels through the collaboration of the large model and multiple small models, and use the integrated soft labels to update the parameters of the anomaly detection model on the edge client;

[0007] S4. Loop the training and update of the anomaly detection model, and detect the real-time collected image data in parallel.

[0008] Further, the configuration of the cloud server containing a large model and the edge client containing an anomaly detection model specifically includes:

[0009] Each production line is configured with an edge client device, and a server is configured in the cloud; the edge client communicates with the cloud server through the network; the edge client deploys a small anomaly detection model with limited resources, and the small anomaly detection model adopts a lightweight convolutional neural network structure, while the cloud server deploys a multi-modal pre-trained large model; the cloud server is responsible for aggregating and optimizing the data and model parameters from multiple clients, enabling the edge small model to obtain the global generalization ability of the large model.

[0010] Further, the specific process of using the camera to collect image data of products on the industrial production line includes: adjusting the shooting frequency of the camera so that the captured pictures can completely cover each product on the production line.

[0011] Further, the training of the anomaly detection model includes: using the cross-entropy loss function to update the parameters of the anomaly detection model.

[0012] Further, the specific process of uploading the parameters of the trained anomaly detection model to the cloud server for aggregation includes: extracting some anomaly detection models, uploading the network parameters among them to the server, and performing weighted averaging on the parameters of the small models through the FedAvg algorithm.

[0013] Further, the specific process of generating integrated soft labels through the cooperation of the large model and multiple small models includes:

[0014] For each sample in the cloud training set: using the large model for forward propagation to generate a prediction distribution; using each selected client small model for forward propagation to generate a prediction distribution; calculating the entropy of the prediction distribution of each selected client small model as the prediction confidence for the current sample; according to the prediction confidence, performing weighted averaging on the prediction distributions of each small model to obtain the integrated prediction of the small models; fusing the large model prediction distribution and the integrated prediction of the small models to form the final integrated soft label.

[0015] Further, the specific process of using the integrated soft label to update the parameters of the anomaly detection model on the edge client includes:

[0016] Taking the aggregated parameters as the parameters of the aggregated small model, adopting the knowledge distillation method, using the integrated soft label as the teacher signal to guide the training of the aggregated small model, updating the parameters of the aggregated small model through gradient descent, sending the updated parameters of the aggregated small model to the edge client, and fusing the local model parameters and the parameters sent from the cloud according to the weight updated by the cloud model.

[0017] Further, the specific process of fusing the local model parameters and the parameters sent from the cloud according to the weight updated by the cloud model is:

[0018] The mean of the feature vectors of all samples in the local dataset of the aggregation small model is extracted on the client side, and then the mean of the feature vectors of all samples in the cloud training set is calculated. The cosine similarity between the mean of the feature vectors of the local dataset and the cloud training set is calculated, and the obtained cosine similarity is normalized and used as the weight of the aggregation parameter to be fused and updated with the parameters of the client small model.

[0019] According to another aspect of the specification, an industrial anomaly detection device based on cloud-edge collaboration is also provided, including a memory and one or more processors. Executable code is stored in the memory, and when the processor executes the executable code, the industrial anomaly detection method based on cloud-edge collaboration as described above is implemented.

[0020] According to another aspect of the specification, a computer-readable storage medium is also provided, on which a program is stored. When the program is executed by a processor, the industrial anomaly detection method based on cloud-edge collaboration as described above is implemented.

[0021] Beneficial effects:

[0022] (1) The method proposed in the present invention deploys a lightweight anomaly detection small model on the edge client for local training and real-time inference, significantly reducing the requirements for the computing power and storage resources of the edge device, effectively solving the problem that the model is too large to run efficiently on resource-constrained edge devices in the background art, and improving the real-time performance of detection.

[0023] (2) The present invention innovatively adopts a knowledge transfer mechanism in which a large model and multiple small models cooperate to generate integrated soft labels. In particular, in the process of generating integrated soft labels, the present invention introduces a dynamic weighting method based on the prediction confidence of each small model for the currently processed sample to integrate the prediction results of multiple small models. The prediction of the dynamically weighted integrated small models is then fused with the prediction of the large model through a preset weight. This mechanism not only balances the global generalization ability of the large model and the local expertise of the small models through the fusion weight, but also enables the integration process to intelligently and adaptively adjust the contributions of each small model for each sample through confidence dynamic weighting, preferentially adopting the opinions of the small models that are more accurate and confident in predicting the current sample, thereby generating higher-quality and more discriminative soft labels. This realizes a more refined and efficient knowledge exchange between the large model and the small models and among the small models themselves, overcomes the limitations of simple averaging or fixed-weight fusion for multi-teacher models in traditional knowledge distillation, significantly enhances the collective wisdom of the edge models, and improves the accuracy and stability of anomaly detection.

[0024] (3) The method proposed in the present invention combines the large-scale and distributed characteristics of the actual industrial production process, and uses a cloud-edge federated learning framework to achieve the co-evolution of model anomaly detection capabilities on multiple production lines. In the model parameter update stage, a dynamic weight allocation mechanism based on the similarity between the local data distribution of the client and the global distribution is introduced. By weighting objectively quantify the degree of dependence of different clients on cloud knowledge, and adaptively fuse local model parameters and cloud migration parameters during parameter update. This design effectively solves the problem of how to balance the global model consistency and local data adaptability in the federated learning environment, and ensures the robustness of the model under diverse working conditions. Description of the Drawings

[0025] Figure 1 is a flowchart of an industrial anomaly detection method based on cloud-edge collaboration provided by an embodiment of the present invention;

[0026] Figure 2 is a system architecture diagram of an industrial product anomaly detection method provided by an embodiment of the present invention;

[0027] Figure 3 is a schematic diagram of an industrial anomaly detection device based on cloud-edge collaboration provided by an embodiment of the present invention. Detailed Embodiment

[0028] The following details the specific implementation method and working principle of the present invention with reference to the accompanying drawings:

[0029] As Figure 1 shown, the present invention proposes an industrial anomaly detection method based on cloud-edge collaboration to achieve efficient and real-time product anomaly detection in the industrial field. Specifically, the method steps are described as follows:

[0030] Step S1, configure a cloud server and edge clients at the industrial site.

[0031] In the industrial site where this embodiment is located, each production line is equipped with an edge client device, and a server is configured in the cloud. The edge client communicates with the cloud server through the network. The edge client deploys a small anomaly detection model with limited resources. The small anomaly detection model adopts a lightweight convolutional neural network structure, and in this embodiment, MobileNetV3 is used to minimize the number of model parameters and computational complexity while ensuring a certain detection accuracy to adapt to the resource limitations of edge devices; the cloud server deploys a large multi-modal pre-trained model, and in this embodiment, the pre-trained model CLIP for vision-language alignment is used. This large model has been pre-trained on a dataset containing a large number of image-text pairs and has powerful cross-modal understanding and feature extraction capabilities, and can provide rich prior knowledge for the small edge model. The cloud server is responsible for aggregating and optimizing the data and model parameters from multiple clients, enabling the small edge model to obtain the global generalization ability of the large model.

[0032] In this embodiment, there are production lines, corresponding to edge clients and small models , and there is 1 cloud server and 1 large multi-modal pre-trained model .

[0033] Step S2: Use a camera to collect image data of products on the industrial production line in real time.

[0034] In this embodiment, the cameras on the production line are responsible for collecting image data of products on the assembly line in real time. According to the product type, production speed, etc. of the production line, the shooting frequency of the camera is adjusted to ensure that the pictures can completely cover each product on the production line. During each model collaborative training, the cameras on the production line will collect a large number of pictures. A relatively small proportion is randomly selected and manually labeled to form image-label pairs , where represents the image, and is the corresponding label. These image-label pairs form the training set . Transmit to the corresponding edge client and the cloud server.

[0035] Step S3: Use the image data to train the anomaly detection model.

[0036] As Figure 2 shown, in the cloud-edge collaboration framework, the anomaly detection model is trained through the following process:

[0037] Step S301: Train the small anomaly detection model on the edge client.

[0038] In this embodiment, each edge client, based on the training set , trains the small model deployed on the client . The update formula for the small model parameters is:

[0039]

[0040] where, is the parameter of the current client model, is the learning rate of the local training of the model, is the cross-entropy loss function, and the formula is:

[0041]

[0042]

[0043] Step S302, the network parameters of the small model are uploaded to the cloud server for aggregation.

[0044] In this embodiment, a part is randomly selected from all the clients to form a set . The following operations will be performed by the clients in the set and the cloud server.

[0045] The edge client uploads the parameters of the trained small model to the cloud server. In the cloud server, the FedAvg algorithm is used to perform weighted averaging on the small model parameters to obtain the aggregated small model , and the formula is:

[0046]

[0047] where, is the parameter of the aggregated small model, is the client 's dataset 's number of samples.

[0048] Step S303, the large model and multiple small models cooperate to generate integrated soft labels and update the aggregated small model.

[0049] In this embodiment, the training sets of the selected clients are all uploaded to the cloud server and merged into the cloud training set. That is:

[0050]

[0051] For each sample in the cloud training set , the following operations are performed to generate integrated soft labels:

[0052] a. Use the large model to perform forward propagation to obtain the predicted distribution ;

[0053] b. Use each selected client small model (i.e., the uploaded model) to perform forward propagation respectively to obtain the predicted distribution ;

[0054] c. Calculate the prediction confidence of each small model. Specifically, for the predicted distribution of the -th small model, its confidence is calculated by the entropy of its predicted distribution. The calculation formula is:

[0055]

[0056] where, represents the set of categories, represents the total number of categories, represents the small model predicting the probability that the current sample belongs to the category .

[0057] d. According to the confidence of each small model for the current sample, perform weighted averaging on its predicted distribution to obtain the small model ensemble prediction:

[0058]

[0059] e. Combine the predicted distribution of the large model with the small model ensemble prediction to form the final ensemble soft label:

[0060]

[0061] where, is the preset combination weight. In this embodiment, can be taken to balance the global knowledge of the large model and the local experience of the small model dynamically integrated based on confidence.

[0062] Adopt the knowledge distillation method and use the ensemble soft label as the teacher signal to guide the training of the aggregated small model . Calculate the knowledge transfer loss:

[0063]

[0064] Update the parameters of the aggregated small model through gradient descent:

[0065]

[0066] Among them, is the updated small model parameter, is the learning rate of knowledge transfer. This process enables the aggregated small model to simultaneously learn the generalization ability of the large model and the local experience of multiple edge small models, realizing the multi-directional exchange of knowledge.

[0067] Step S304: Redistribute the small model after knowledge transfer to the edge client for network parameter update.

[0068] In this embodiment, the cloud server distributes the small model parameter to the edge clients in the set . The client fuses the local model parameter and the parameter sent from the cloud according to the objectively calculated weight of the cloud model update. The calculation method of the weight is as follows:

[0069] a. Feature extraction: Use the current aggregated small model to extract the feature vectors of all samples in the client's local dataset .

[0070] b. Calculate the feature mean: Calculate the mean of the feature vectors of the client's local dataset.

[0071] c. Calculate the global feature mean: Calculate the mean of the feature vectors of all samples in the cloud training set .

[0072] d. Calculate the cosine similarity: Calculate the cosine similarity between and :

[0073]

[0074] e. Obtain the normalized weight:

[0075]

[0076] The client updates its model parameter:

[0077]

[0078] In this embodiment, the above steps S301 to S304 are continuously looped, and the small model of the edge client is continuously optimized through the collaborative knowledge transfer mechanism, improving the anomaly detection performance. During this process, the cloud server can dynamically adjust the frequency of executing cloud-edge collaborative training and the fusion weight Hyperparameters related to model updates.

[0079] Step S4: Use the anomaly detection model to detect the image data.

[0080] In this embodiment, the edge client uses the current small model to infer the real-time collected image data and generate an anomaly detection result. The inference process is parallel to the model training to ensure the real-time requirements on the production line. When the small model is updated, the new model parameters will be immediately applied to the inference task to provide real-time feedback to the production line operators.

[0081] Corresponding to the foregoing embodiment of an industrial anomaly detection method based on cloud-edge collaboration, the present invention also provides an embodiment of an industrial anomaly detection device based on cloud-edge collaboration.

[0082] See Figure 3 , an industrial anomaly detection device based on cloud-edge collaboration provided by an embodiment of the present invention includes a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it is used to implement an industrial anomaly detection method in the foregoing embodiment.

[0083] The embodiment of the industrial anomaly detection device provided by the present invention can be applied to any device with data processing capabilities. The any device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logically meaningful device, it is formed by the processor of any device with data processing capabilities reading the corresponding computer program instructions in the non-volatile memory into the memory for operation. From the hardware level, as Figure 3 shown, it is a hardware structure diagram of any device with data processing capabilities where the industrial anomaly detection device provided by the present invention is located. Except for Figure 3 the processor, memory, network interface, and non-volatile memory shown, the any device with data processing capabilities where the device in the embodiment is located usually also includes other hardware according to the actual functions of the any device with data processing capabilities, which will not be elaborated here.

[0084] The implementation processes of the functions and roles of each unit in the above device are specifically detailed in the implementation processes of the corresponding steps in the above method, and will not be elaborated here.

[0085] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions in the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0086] An embodiment of the present invention also provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it implements a method for industrial anomaly detection based on cloud-edge collaboration in the above embodiments.

[0087] The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store the data that has been output or will be output.

[0088] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for industrial anomaly detection based on cloud-edge collaboration described above.

[0089] Those skilled in the art will readily think of other embodiments of the present application after considering the specification and practicing the content disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the claims.

[0090] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and do not limit the present application. The present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. An industrial anomaly detection method based on cloud-edge collaboration, characterized in that It includes the following steps: S1. Configure a cloud server containing a large model and an edge client containing an anomaly detection model in the industrial field; S2. Use a camera to collect and annotate the image data of products on the industrial production line in real time; S3. Use the image data to train the anomaly detection model on the edge client; upload the parameters of the trained anomaly detection model to the cloud server for aggregation, generate an integrated soft label through the cooperation of the large model and multiple small models, and use the integrated soft label to update the parameters of the anomaly detection model on the edge client; S4. Cycle the training and update of the anomaly detection model, and detect the real-time collected image data in parallel.

2. The industrial anomaly detection method based on cloud-edge collaboration according to claim 1, wherein The configuration of the cloud server containing the large model and the edge client containing the anomaly detection model specifically includes: Configure an edge client device for each production line and configure a server in the cloud; the edge client communicates with the cloud server through the network; the edge client deploys a small anomaly detection model with limited resources, and the small anomaly detection model adopts a lightweight convolutional neural network structure, while the cloud server deploys a multi-modal pre-trained large model; the cloud server is responsible for aggregating and optimizing the data and model parameters from multiple clients, enabling the edge small model to obtain the global generalization ability of the large model.

3. The industrial anomaly detection method based on cloud-edge collaboration according to claim 1, wherein, The use of the camera to collect the image data of products on the industrial production line in real time specifically includes: adjusting the shooting frequency of the camera so that the captured pictures can completely cover each product on the production line.

4. The industrial anomaly detection method based on cloud-edge collaboration according to claim 1, characterized in that, The training of the anomaly detection model includes: using the cross-entropy loss function to update the parameters of the anomaly detection model.

5. An industrial anomaly detection method based on cloud-edge collaboration according to claim 1, characterized in that The uploading of the parameters of the trained anomaly detection model to the cloud server for aggregation specifically includes: extracting some anomaly detection models, uploading the network parameters therein to the server, and performing weighted averaging on the parameters of the small models through the FedAvg algorithm.

6. The industrial anomaly detection method based on cloud-edge collaboration according to claim 1, wherein, The generation of the integrated soft label through the cooperation of the large model and multiple small models specifically includes: For each sample in the cloud training set: use the large model for forward propagation to generate a prediction distribution; use each selected client small model for forward propagation to generate a prediction distribution; calculate the entropy of the prediction distribution of each selected client small model as the prediction confidence for the current sample; according to the prediction confidence, perform weighted averaging on the prediction distributions of each small model to obtain the integrated prediction of the small models; fuse the large model prediction distribution and the integrated prediction of the small models to form the final integrated soft label.

7. A method for industrial anomaly detection based on cloud-edge collaboration according to claim 1, characterized in that, The use of the integrated soft label to update the parameters of the anomaly detection model on the edge client includes: Taking the aggregated parameters as the parameters of the aggregated small model, adopting the knowledge distillation method, using the integrated soft label as the teacher signal to guide the training of the aggregated small model, updating the parameters of the aggregated small model through gradient descent, sending the updated parameters of the aggregated small model to the edge client, and fusing the local model parameters and the parameters sent from the cloud according to the cloud model update weight.

8. The industrial anomaly detection method based on cloud-edge collaboration according to claim 7, wherein, The specific method of fusing the local model parameters and the parameters sent from the cloud according to the cloud model update weight is: The mean of the feature vectors of all samples in the client's local dataset is extracted by the aggregated small model, and then the mean of the feature vectors of all samples in the cloud training set is calculated. The cosine similarity between the mean of the feature vectors of the local dataset and the cloud training set is calculated, and the obtained cosine similarity is normalized and used as the weight of the aggregation parameter to be fused and updated with the parameters of the client small model.

9. An industrial anomaly detection device based on cloud-edge collaboration, comprising a memory and one or more processors, wherein executable code is stored in the memory, characterized in that, When the processor executes the executable code, it implements a cloud-edge collaborative industrial anomaly detection method as described in any one of claims 1-8.

10. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements a cloud-edge collaborative industrial anomaly detection method as described in any one of claims 1-8.

Images