Intelligent industrial quality inspection method and system based on AI visual identification

Abstract

The invention provides an intelligent industrial quality inspection method and system based on AI visual identification, and relates to the technical field of visual identification, and the method comprises the steps: inputting an industrial product image collected at an edge end into a pre-trained defect region detection model, and obtaining a defect region detection result; extracting defect area features according to a defect area detection result, carrying out dimension reduction processing to obtain compressed defect area features, and deciding to unload a defect classification task to a cloud end or directly carry out defect classification at the edge end according to an adaptive task unloading strategy in combination with working data of the edge equipment; if the defect classification task is determined to be unloaded to the cloud, transmitting the compressed defect area features to the cloud through a network, performing defect type identification on the cloud by adopting a pre-trained defect classification model, outputting a classification result of the defect area, returning the classification result to an edge end, and combining a defect area detection result and the classification result of the defect area to obtain the defect classification task. And generating a defect visualization effect picture and a quality inspection report.

Description

Technical Field

[0001] The present invention relates to visual recognition technology, and in particular to an intelligent industrial quality inspection method and system based on AI visual recognition. Background Art

[0002] With the continuous improvement of the level of industrial automation, the requirements for product quality in industrial production are becoming more and more stringent. Traditional industrial quality inspection mainly relies on manual inspection, which has problems such as low efficiency, low accuracy, and strong subjectivity. In order to solve these problems, various automated quality inspection technologies have emerged, among which quality inspection technology based on machine vision has been widely used due to its advantages such as non-contact, high efficiency, and objectivity. However, traditional machine vision quality inspection technology mainly relies on manually designed feature extraction and defect recognition algorithms, facing challenges such as complex algorithm design, poor adaptability, and weak generalization ability. In recent years, the rapid development of artificial intelligence technology has brought new opportunities to the field of industrial quality inspection. AI visual recognition technology based on deep learning overcomes the limitations of traditional methods by automatically learning feature representations in data, and shows great application potential in industrial quality inspection.

[0003] Although AI-based visual recognition-based intelligent industrial quality inspection technology has made some progress, there are still some challenges and shortcomings, such as fine identification and classification of defects, adaptability to complex industrial environments, and how to improve the efficiency and performance of the model. Therefore, it is urgent to further research and develop innovative AI visual recognition technology to meet the high standards of industrial quality inspection. Summary of the invention

[0004] The embodiments of the present invention provide an intelligent industrial quality inspection method and system based on AI visual recognition, which can solve the problems in the prior art.

[0005] According to a first aspect of the embodiments of the present invention,

[0006] Provide an intelligent industrial quality inspection method based on AI visual recognition, including:

[0007] At the edge, the collected industrial product images are input into the pre-trained defect area detection model, multiple candidate defect areas are generated through the region proposal network, and defect judgment and bounding box regression are performed on each candidate defect area through the fully connected layer to obtain the defect area detection result;

[0008] Extract defect area features based on defect area detection results, perform dimensionality reduction processing to obtain compressed defect area features, and decide to offload defect classification tasks to the cloud or perform defect classification directly on the edge based on the adaptive task offloading strategy and the working data of the edge device;

[0009] If it is decided to offload the defect classification task to the cloud, the compressed defect area features will be transmitted to the cloud through the network, and the pre-trained defect classification model will be used in the cloud to identify the defect type, output the classification results of the defect area, and return the classification results to the edge. Combined with the defect area detection results and the defect area classification results, a defect visualization effect diagram and quality inspection report are generated.

[0010] In an optional embodiment,

[0011] Before the collected industrial product images are input into the pre-trained defect area detection model at the edge, the following steps are also included:

[0012] Training the defect area detection model, for each training iteration, randomly selecting a batch of image samples to be trained from the training set, randomly perturbing the pixel value of each image sample to be trained, generating an initial adversarial sample, and iteratively optimizing the initial adversarial sample;

[0013] In each iteration step, the gradient of the adversarial loss function of the current adversarial sample on the defect area detection model is calculated and normalized to unit length. The current perturbation is updated according to the normalized gradient, and the updated perturbation is added to the original image sample to be trained to obtain a new adversarial sample. The iteration is repeated until the preset conditions are met and the target adversarial sample is output.

[0014] The output target adversarial sample and the original image sample to be trained are input into the defect area detection model for training, and the first loss function of the target adversarial sample and the second loss function of the original image to be trained are calculated respectively, and the first loss function and the second loss function are fused to obtain the comprehensive training loss;

[0015] According to the comprehensive training loss, the parameters of the defect area detection model are updated using a parameter updating algorithm, and the parameter updating steps of the defect area detection model are repeated until a preset termination condition is met to obtain a trained defect area detection model.

[0016] In an optional embodiment,

[0017] The calculation formula for updating the parameters of the defect area detection model using the parameter updating algorithm is as follows:

[0018]

[0019] Among them, θ * represents the updated parameters, θ represents the initial model parameters, λ represents the learning rate, β1 represents the exponential decay rate of the control first-order moment estimate, m represents the first-order moment estimate, represents the gradient of the comprehensive training loss to the model parameters, represents the attenuation coefficient of the first-order moment estimate in the tth iteration, t represents the current number of iterations, β2 represents the exponential attenuation rate of the second-order moment estimate, v represents the second-order moment estimate, represents the attenuation coefficient of the second-order moment estimate in the tth iteration, and ε represents a constant parameter.

[0020] In an optional embodiment,

[0021] According to the adaptive task offloading strategy and combined with the working data of the edge device, it is decided to offload the defect classification task to the cloud or directly perform defect classification on the edge, including:

[0022] The task offloading problem is modeled as a Markov decision process, and a state space, an action space, and a reward function are defined. The state space includes the edge device state, the network state, and the task state. The action space includes two actions: executing the defect classification task at the local edge or offloading the defect classification task to the cloud. The reward function comprehensively considers the task execution time, energy consumption, and network overhead.

[0023] A deep reinforcement learning algorithm is used to solve the Markov decision process, calculate the learning objectives of the initial value function network, and update the initial value function network parameters by minimizing the task offloading loss. The iteration is repeated until the preset termination condition is met, and finally the optimal offloading strategy is output;

[0024] When the actual system is running, the current state observation value of the system is obtained in real time and input into the Markov decision process to obtain the optimal offloading strategy. According to the optimal offloading strategy, the defect classification task is executed locally or offloaded to the cloud. The state observation value includes the computing resources and energy status of the edge device, the network status and the characteristics of the task to be processed.

[0025] In an optional embodiment,

[0026] The method further comprises:

[0027] If it is decided to perform defect classification at the edge, a lightweight convolutional neural network is designed at the edge as the first classification model, and a defect dataset is constructed based on the compressed defect area features. The defect classification model deployed in the cloud is used as the second classification model.

[0028] Using the second classification model to infer the defect data set, obtain soft labels for each sample data on different defect categories, wherein the soft labels include prediction confidence information of the second classification model for the sample category;

[0029] The soft labels of the second classification model are used as additional supervisory signals to train the first classification model. The comprehensive classification loss function of the first classification model is obtained by calculating the cross entropy loss between the prediction results of the first classification model and the true value, and the divergence between the prediction results of the first classification model and the soft labels of the second classification model. The final trained first classification model is obtained by minimizing the comprehensive classification loss function.

[0030] The compressed defect area features are input into the trained first classification model, and the classification results of the defect area are output.

[0031] In an optional embodiment,

[0032] The calculation formula of the comprehensive classification loss function is as follows:

[0033]

[0034] Where L represents the comprehensive classification loss function, N represents the sample set, i represents the sample index, α represents the weight coefficient for balancing the true loss and the soft label loss, CE(·) represents the cross entropy loss between the prediction result of the first classification model and the true value, and y i represents the true category label corresponding to the i-th sample, represents the predicted probability of the first classification model for the i-th sample, S represents the first classification model, τ represents the parameter that controls the smoothness of the soft label, KL(·) represents the divergence between the prediction result of the first classification model and the soft label of the second classification model, represents the soft label of the second classification model of the i-th sample, and T represents the second classification model.

[0035] In an optional embodiment,

[0036] The compressed defect area features are transmitted to the cloud through the network, and the pre-trained defect classification model is used in the cloud to identify the defect type. The classification results of the defect area are output including:

[0037] The defective regions are regarded as nodes in the graph structure. The features of the nodes include the image attributes of the defective regions. The spatial adjacency between the defective regions is regarded as the edge. The weight of the edge reflects the similarity between the defective regions to construct the graph structure.

[0038] Construct a graph convolutional neural network, including a graph convolution layer, a pooling layer, and a classification layer. The graph convolution layer receives the node feature matrix and the adjacency matrix of the graph output by the previous layer, and updates the feature representation of the node by aggregating the neighborhood information of each node and applying the convolution operation. The attention pooling layer adaptively aggregates the node features output by the graph convolution layer by learning the importance weight of each node to obtain the feature representation at the graph level. The classification layer inputs the graph-level feature representation output by the attention pooling layer into the fully connected neural network to generate the probability distribution of the defect category.

[0039] The compressed defect area features are converted into a graph structure representation and input into the trained graph convolutional neural network to obtain the predicted probability distribution of the defect category, and the highest category in the predicted probability distribution is selected as the final classification result.

[0040] According to a second aspect of the embodiments of the present invention,

[0041] Through an intelligent industrial quality inspection system based on AI visual recognition, including:

[0042] The first unit is used to input the collected industrial product image into the pre-trained defect area detection model at the edge, generate multiple candidate defect areas through the region proposal network, and perform defect judgment and bounding box regression on each candidate defect area through the fully connected layer to obtain the defect area detection result;

[0043] The second unit is used to extract defect area features based on the defect area detection results, and perform dimensionality reduction processing to obtain compressed defect area features. According to the adaptive task offloading strategy and the working data of the edge device, it is decided to offload the defect classification task to the cloud or directly perform defect classification on the edge.

[0044] The third unit is used to transmit the compressed defect area features to the cloud through the network if it is decided to offload the defect classification task to the cloud, use the pre-trained defect classification model in the cloud to identify the defect type, output the classification result of the defect area, and return the classification result to the edge end. Combined with the defect area detection result and the defect area classification result, a defect visualization effect diagram and a quality inspection report are generated.

[0045] According to a third aspect of the embodiments of the present invention,

[0046] An electronic device is provided, comprising:

[0047] processor;

[0048] a memory for storing processor-executable instructions;

[0049] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0050] A fourth aspect of the embodiments of the present invention is:

[0051] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0052] In this embodiment, by adopting a deep learning target detection model, the defect area in the industrial product image can be accurately detected, and the bounding box coordinates of each defect area are given to achieve high-precision positioning of the defect. By extracting features and reducing the dimension of the detected defect area, the compressed defect features can be obtained, which reduces the overhead of network transmission and cloud computing and improves the system efficiency. Adopting an adaptive task offloading strategy, it is dynamically determined whether to classify defects at the edge or in the cloud according to the working status of the edge device, which realizes the collaborative work of the cloud and the edge, and improves the flexibility and scalability of the system. Using a pre-trained defect classification model in the cloud to classify the compressed defect features, the specific type of defects can be identified with high precision, providing an important basis for subsequent quality analysis and decision-making. By offloading the computationally intensive defect classification task to the cloud, the computing pressure of the edge device is reduced, and at the same time, the powerful computing power of the cloud is utilized to achieve efficient collaborative computing between the edge and the cloud, and at the same time, efficient detection, classification and visualization of industrial product defects are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 This is a flow chart of an intelligent industrial quality inspection method based on AI visual recognition according to an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram of the structure of an intelligent industrial quality inspection system based on AI visual recognition according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0057] Figure 1FIG. 1 is a flow chart of an intelligent industrial quality inspection method based on AI visual recognition according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0058] S101. Input the collected industrial product image into the pre-trained defect area detection model at the edge, generate multiple candidate defect areas through the region proposal network, and perform defect judgment and bounding box regression on each candidate defect area through the fully connected layer to obtain the defect area detection result;

[0059] Exemplarily, first, a high-definition camera or an industrial camera is used at the edge to collect image data of the industrial product to be inspected, and necessary preprocessing operations are performed, such as image denoising, enhancement, and normalization, to improve image quality and feature expression capabilities.

[0060] The feature extraction part of the defect area detection model usually uses a pre-trained convolutional neural network as the backbone network, such as ResNet, VGG, MobileNet, etc. The backbone network is pre-trained on large-scale datasets such as ImageNet to learn rich visual feature representations, which can significantly accelerate the convergence of the model and improve the detection performance. Region proposals and defect judgments are performed on the feature map output by the last convolution block (Conv Block) of the backbone network.

[0061] The region proposal network slides a 3x3 convolution kernel on the convolution feature map extracted by the backbone network to generate candidate regions at the corresponding positions. For each candidate region, RPN performs binary classification (foreground / background) and bounding box regression through two parallel fully connected layer branches. The binary classification branch predicts whether the candidate region contains defects, and the bounding box regression branch predicts the offset of the candidate region relative to the predefined Anchor Box, which is used to adjust the position and size of the candidate region. The training of RPN is end-to-end optimized through a weighted combination of binary classification loss and bounding box regression loss.

[0062] For each candidate defect region generated by RPN, RoI pooling or RoI alignment is used to extract the fixed-length feature vector of the corresponding region on the convolutional feature map. The extracted feature vector is then input into the classifier and regressor composed of the fully connected layer to perform defect category judgment and precise bounding box regression, respectively, to obtain the defect detection results for each candidate region.

[0063] The detection results of all candidate areas are post-processed, such as non-maximum suppression to remove redundant and overlapping detection frames, to obtain the final defect area detection results.

[0064] In this embodiment,

[0065] In an optional embodiment,

[0066] Before the collected industrial product images are input into the pre-trained defect area detection model at the edge, the following steps are also included:

[0067] Training the defect area detection model, for each training iteration, randomly selecting a batch of image samples to be trained from the training set, randomly perturbing the pixel value of each image sample to be trained, generating an initial adversarial sample, and iteratively optimizing the initial adversarial sample;

[0068] In each iteration step, the gradient of the adversarial loss function of the current adversarial sample on the defect area detection model is calculated and normalized to unit length. The current perturbation is updated according to the normalized gradient, and the updated perturbation is added to the original image sample to be trained to obtain a new adversarial sample. The iteration is repeated until the preset conditions are met and the target adversarial sample is output.

[0069] The output target adversarial sample and the original image sample to be trained are input into the defect area detection model for training, and the first loss function of the target adversarial sample and the second loss function of the original image to be trained are calculated respectively, and the first loss function and the second loss function are fused to obtain the comprehensive training loss;

[0070] According to the comprehensive training loss, the parameters of the defect area detection model are updated using a parameter updating algorithm, and the parameter updating steps of the defect area detection model are repeated until a preset termination condition is met to obtain a trained defect area detection model.

[0071] Among them, common adversarial sample generation algorithms include the fast gradient sign method (FGSM) and projected gradient descent (PGD). FGSM generates adversarial samples by calculating the gradient of the loss function with respect to the input image and adding perturbations along the sign direction of the gradient. PGD generates more powerful adversarial samples based on FGSM through multi-step iterative optimization.

[0072] Exemplarily, for each training iteration, a batch of image samples to be trained are randomly selected from the training set, and the pixel values ​​of each image sample to be trained are randomly perturbed to generate an initial adversarial sample. Specifically, a random noise matrix of the same size as the original image sample to be trained is first generated, and its pixel values ​​are randomly initialized within a certain range, such as sampling on a uniform distribution or Gaussian distribution between [0, 255]. The original image is added to the perturbation matrix to obtain the initial adversarial sample.

[0073] Set the number of iterations to control the number of iterations of optimization. In each iteration step, input the current adversarial sample into the defect area detection model and calculate the loss function of the current adversarial sample on the defect area detection model. Use the back propagation algorithm to calculate the gradient of the current adversarial sample on the defect area detection model with respect to the input image. Normalize the calculated gradient to unit length, that is, divide the gradient by its L2 norm.

[0074] According to the normalized gradient, the current perturbation is updated. The update can be performed using gradient ascent or gradient descent. In order to control the range of the perturbation, the updated perturbation can be restricted, for example, using the L∞ norm restriction, that is, restricting the perturbation to a preset range.

[0075] The updated perturbation is added to the original image to obtain a new adversarial sample. The process of iterative optimization and updating the perturbation is repeated until the preset number of iterations is reached to generate the target adversarial sample.

[0076] The output target adversarial sample and the original image sample to be trained are input into the defect area detection model for training, and the first loss function of the target adversarial sample and the second loss function of the original image to be trained are calculated respectively, wherein the first loss function and the second loss function can adopt binary cross entropy loss, and optionally, focus loss can also be adopted.

[0077] The loss functions of the original image and the adversarial sample are weighted summed to obtain the total training loss. When weighted summing, the weight ratio of the original sample and the adversarial sample can be adjusted according to actual needs to balance the performance of the model on clean samples and adversarial samples.

[0078] According to the total training loss, the parameters of the defect area detection model are updated using the gradient descent optimization algorithm. The gradient of the loss function with respect to the model parameters is calculated through the back propagation algorithm, and the optimization direction of the parameters is guided according to the gradient, so that the model can adapt to both the original samples and the adversarial samples. Specifically, the parameters are initialized first, the gradient of the loss function with respect to the model parameters is calculated in each iteration, the first-order moment estimation and the second-order moment estimation are updated, and the deviation in the initial iteration is corrected according to the attenuation coefficient of the first-order moment estimation and the attenuation coefficient of the second-order moment estimation. The parameters of the defect area detection model are updated based on the first-order moment estimation, the second-order moment estimation and the deviation value.

[0079] Repeat the above steps until the performance indicators of the model on the validation set (such as F1 value, mAP, etc.) reach a satisfactory level, or reach the preset number of training rounds, and obtain a trained defect area detection model.

[0080] In an optional embodiment,

[0081] The calculation formula for updating the parameters of the defect area detection model using the parameter updating algorithm is as follows:

[0082]

[0083] Among them, θ * represents the updated parameters, θ represents the initial model parameters, λ represents the learning rate, β1 represents the exponential decay rate of the control first-order moment estimate, m represents the first-order moment estimate, represents the gradient of the comprehensive training loss to the model parameters, represents the attenuation coefficient of the first-order moment estimate in the tth iteration, t represents the current number of iterations, β2 represents the exponential attenuation rate of the second-order moment estimate, v represents the second-order moment estimate, represents the attenuation coefficient of the second-order moment estimate in the tth iteration, and ε represents a constant parameter.

[0084] In this embodiment, by adding adversarial samples during training, the robustness of the model to noise and small disturbances can be enhanced. Adversarial samples simulate some disturbances that are imperceptible to humans but have an impact on the model. Using adversarial samples during training can enable the model to learn these disturbances, thereby improving its tolerance to noise and small disturbances. By integrating the comprehensive training loss of adversarial sample loss and normal sample loss, the model can be prompted to learn the features of adversarial samples while learning the features of normal samples, thereby improving the model's detection accuracy for defects in various situations. During the adversarial training process, the model needs to learn the features of normal samples and adversarial samples at the same time, which requires the model to extract richer features, thereby enhancing the classification ability of the model.

[0085] S102. Extract defect area features based on defect area detection results, perform dimensionality reduction processing to obtain compressed defect area features, and decide to offload the defect classification task to the cloud or perform defect classification directly on the edge according to the adaptive task offloading strategy and the working data of the edge device;

[0086] Among them, the dimension reduction processing of the defect area features can be carried out by principal component analysis and other methods, while retaining most of the feature information, reducing the dimension of the feature vector to obtain a compressed feature vector. The working data of the edge device includes the computing power of the edge device, the network bandwidth, and the computing amount and data volume of the task.

[0087] In an optional embodiment,

[0088] According to the adaptive task offloading strategy and combined with the working data of the edge device, it is decided to offload the defect classification task to the cloud or directly perform defect classification on the edge, including:

[0089] The task offloading problem is modeled as a Markov decision process, and a state space, an action space, and a reward function are defined. The state space includes the edge device state, the network state, and the task state. The action space includes two actions: executing the defect classification task at the local edge or offloading the defect classification task to the cloud. The reward function comprehensively considers the task execution time, energy consumption, and network overhead.

[0090] A deep reinforcement learning algorithm is used to solve the Markov decision process, calculate the learning objectives of the initial value function network, and update the initial value function network parameters by minimizing the task offloading loss. The iteration is repeated until the preset termination condition is met, and finally the optimal offloading strategy is output;

[0091] When the actual system is running, the current state observation value of the system is obtained in real time and input into the Markov decision process to obtain the optimal offloading strategy. According to the optimal offloading strategy, the defect classification task is executed locally or offloaded to the cloud. The state observation value includes the computing resources and energy status of the edge device, the network status and the characteristics of the task to be processed.

[0092] Exemplarily, first, the task offloading problem is formalized as a Markov decision process (MDP), and its state space, action space and reward function are defined. The state space includes the edge device state, network state and task state, including CPU utilization, memory occupancy, power, etc., reflecting the ability and duration of the device to perform tasks. The network state represents the network status between the edge device and the cloud, including network bandwidth, delay, jitter, etc., reflecting the transmission overhead and delay of task offloading. The task state represents the characteristics of the current task to be processed, such as the computational amount, data volume, priority, etc. of the task, reflecting the task's demand for computing resources and transmission resources.

[0093] The action space includes two actions: performing the defect classification task locally on the edge or offloading the defect classification task to the cloud. The agent needs to choose an action based on the current state to optimize the long-term benefit.

[0094] The reward function defines the immediate reward value obtained by taking the corresponding action in a certain state, which is used to guide the agent to learn the optimal strategy. The reward function is designed by comprehensively considering factors such as task execution time, energy consumption and network overhead, and different weight coefficients are set to balance the contribution of different factors to the reward.

[0095] Based on the MDP model, a deep reinforcement learning algorithm is used to solve the optimal unloading strategy, such as deep Q network, deep deterministic policy gradient, etc., and initialize the corresponding neural network structure. Convolutional neural network or recurrent neural network is usually used to process state information, and a fully connected layer is used to estimate the action value function or policy function.

[0096] The agent (unloading decision maker) interacts with the environment (unloading system), selects actions based on the current state, executes unloading decisions, and observes the next state and reward of the environment feedback. The collected experience data (state, action, reward, next state) is stored in the experience replay buffer for subsequent model training.

[0097] A small batch of experience data is randomly sampled from the experience replay buffer as training samples. For each sampled experience data, a target policy is calculated, which is the optimal action calculated based on the next state and reward.

[0098] According to the difference between the strategy predicted by the current model and the target strategy, a loss function is constructed. Usually, the mean square error loss function or the cross entropy loss function is used. The gradient descent algorithm is used to calculate the gradient of the loss function to the model parameters through back propagation, and the model parameters are updated so that the predicted value gradually approaches the target value.

[0099] Repeat the process of interacting with the environment, sampling empirical data, calculating target values, calculating loss functions, and updating model parameters, and continuously iterate and optimize the model until the preset convergence conditions are met, such as reaching the maximum number of training rounds or the strategy performance is stable.

[0100] The trained deep reinforcement learning model is deployed to the actual offloading decision system. According to the current system status, the model is used to predict the optimal offloading decision and perform corresponding actions to realize intelligent task offloading.

[0101] In this embodiment, by acquiring the system state observation value in real time and making decisions based on the optimal strategy obtained through training, a decision can be made dynamically whether to execute locally or offload to the cloud based on the real-time computing resources, energy status and network status of the edge device, thereby achieving the adaptability of task offloading. By modeling the task offloading problem as a Markov decision process and solving it with a deep reinforcement learning algorithm, a global optimal offloading strategy that can maximize the overall benefit in long-term operation can be obtained. By integrating multiple influencing factors such as edge device status, network status and task status into the state space, and weighing various overheads through a reward function, the final task offloading decision can comprehensively consider the impact of multiple system states, and can achieve dynamic adaptive and globally optimal task offloading decisions, thereby improving resource utilization efficiency.

[0102] In an optional embodiment,

[0103] The method further comprises:

[0104] If it is decided to perform defect classification at the edge, a lightweight convolutional neural network is designed at the edge as the first classification model, and a defect dataset is constructed based on the compressed defect area features. The defect classification model deployed in the cloud is used as the second classification model.

[0105] Using the second classification model to infer the defect data set, obtain soft labels for each sample data on different defect categories, wherein the soft labels include prediction confidence information of the second classification model for the sample category;

[0106] The soft labels of the second classification model are used as additional supervisory signals to train the first classification model. The comprehensive classification loss function of the first classification model is obtained by calculating the cross entropy loss between the prediction results of the first classification model and the true value, and the divergence between the prediction results of the first classification model and the soft labels of the second classification model. The final trained first classification model is obtained by minimizing the comprehensive classification loss function.

[0107] The compressed defect area features are input into the trained first classification model, and the classification results of the defect area are output.

[0108] Exemplarily, a lightweight first classification model is first designed, such as MobileNetV3 or ShuffleNetV2, for deployment on the edge. The network structure of the first classification model should minimize the number of parameters and computational complexity of the model while meeting the accuracy requirements. At the same time, the defect classification model deployed in the cloud is used as the second classification model, and the second classification model is used to infer the training set to obtain soft labels, i.e., probability distributions, for each sample in different defect categories. The soft label contains the prediction confidence information of the first classification model for the sample category, which can provide richer knowledge for the student model compared to the hard label (i.e., the true category of the sample). Specifically, the second classification model is used to predict the training samples to obtain the probability distribution of each sample belonging to each category. The probability distribution is converted into a soft label, that is, the probability distribution is used as the label of each category to guide the training of the first classification model.

[0109] The soft labels of the second classification model are used as additional supervisory signals to guide the training of the first classification model. Specifically, the training objective of the first classification model consists of two parts: one is to minimize the cross entropy loss between the prediction results of the first classification model and the true hard labels, and the other is to minimize the KL divergence between the prediction results of the first classification model and the soft labels of the second classification model. By jointly optimizing the hard label loss and the soft label loss, the first classification model can simultaneously learn the true category information of the sample and the knowledge of the second classification model.

[0110] During the training process, the parameters of the second classification model are fixed and only the parameters of the first classification model are updated, and the model pruning and quantization technology is introduced to further compress the first classification model. Specifically, in the model pruning stage, by analyzing the importance of each connection in the model, redundant or unimportant connections are removed, thereby reducing the number of model parameters. Common pruning methods include threshold pruning based on weight size, sensitivity pruning based on gradient, etc. In the model quantization stage, the model parameters originally represented by 32-bit floating point numbers are quantized to fixed-point numbers with lower bit widths (such as 8 bits or 16 bits). By reducing the bit width of the parameters, the storage size of the model can be significantly reduced, and the calculation operations in the reasoning process can be accelerated. The quantization method can use uniform quantization or non-uniform quantization, and be optimized according to data distribution and hardware characteristics. The quantized model parameters need to be re-trained to compensate for the accuracy loss caused by quantization. The quantization-aware training method is adopted to simulate the quantization operation in the forward propagation process, and the model parameters are updated in the back propagation process, so that the performance of the quantized model is as close to the original model as possible.

[0111] Finally, a lightweight first classification model is obtained after knowledge distillation, model pruning and quantization. The first classification model is deployed on the edge device for online industrial defect classification. The compressed first classification model has lower latency and smaller memory usage at the edge. The second classification model is continuously trained with new data in the cloud, and the first classification model is regularly updated and maintained.

[0112] In an optional embodiment,

[0113] The calculation formula of the comprehensive classification loss function is as follows:

[0114]

[0115] Where L represents the comprehensive classification loss function, N represents the sample set, i represents the sample index, α represents the weight coefficient for balancing the true loss and the soft label loss, CE(·) represents the cross entropy loss between the prediction result of the first classification model and the true value, and y i represents the true category label corresponding to the i-th sample, represents the predicted probability of the first classification model for the i-th sample, S represents the first classification model, τ represents the parameter that controls the smoothness of the soft label, KL(·) represents the divergence between the prediction result of the first classification model and the soft label of the second classification model, represents the soft label of the second classification model of the i-th sample, and T represents the second classification model.

[0116] In this embodiment, by designing a lightweight convolutional neural network as the first classification model, it can run efficiently on edge devices with limited computing resources to meet the low power consumption and low latency requirements of the edge. The large defect classification model deployed in the cloud is used as the second classification model, and the soft labels of its prediction results are used as additional supervision signals. The classification knowledge of the cloud model can be migrated to the lightweight model at the edge to improve the classification ability of the edge model. By calculating the loss of the first classification model with the true label and the loss with the soft label of the second classification model, and minimizing the combination of these two parts of the loss, the training effect of the first classification model at the edge on a limited data set can be improved, thereby improving its classification accuracy. By deploying a lightweight model at the edge and using the cloud model knowledge migration method, efficient defect classification at the edge is achieved.

[0117] S103. If it is decided to offload the defect classification task to the cloud, the compressed defect area features are transmitted to the cloud through the network, and the pre-trained defect classification model is used in the cloud to identify the defect type, output the classification result of the defect area, and return the classification result to the edge. Combined with the defect area detection result and the defect area classification result, a defect visualization effect diagram and a quality inspection report are generated.

[0118] Whether defect classification and analysis is completed in the cloud or on the edge, we will match the analysis results with the defect areas detected in the original image to generate a final defect detection report. The report not only includes the location and type of the defect, but also gives an assessment of the severity of the defect, providing a reference for subsequent product quality improvements.

[0119] Optionally, a feedback mechanism can be established between the cloud and the edge. Specifically, after receiving the compressed feature vector uploaded by the edge, the cloud can not only classify and analyze defects, but also evaluate the effects of feature compression and task offloading strategies. If it is found that the feature loss after compression is too much or the task offloading decision is unreasonable, feedback information can be sent to the edge to dynamically adjust relevant parameters to optimize the performance of the entire system.

[0120] In an optional embodiment,

[0121] The compressed defect area features are transmitted to the cloud through the network, and the pre-trained defect classification model is used in the cloud to identify the defect type. The classification results of the defect area are output including:

[0122] The defective regions are regarded as nodes in the graph structure. The features of the nodes include the image attributes of the defective regions. The spatial adjacency between the defective regions is regarded as the edge. The weight of the edge reflects the similarity between the defective regions to construct the graph structure.

[0123] Construct a graph convolutional neural network, including a graph convolution layer, a pooling layer, and a classification layer. The graph convolution layer receives the node feature matrix and the adjacency matrix of the graph output by the previous layer, and updates the feature representation of the node by aggregating the neighborhood information of each node and applying the convolution operation. The attention pooling layer adaptively aggregates the node features output by the graph convolution layer by learning the importance weight of each node to obtain the feature representation at the graph level. The classification layer inputs the graph-level feature representation output by the attention pooling layer into the fully connected neural network to generate the probability distribution of the defect category.

[0124] The compressed defect area features are converted into a graph structure representation and input into the trained graph convolutional neural network to obtain the predicted probability distribution of the defect category, and the highest category in the predicted probability distribution is selected as the final classification result.

[0125] For example, each defect is considered as a node in the graph, and the relationship between defects is considered as the edge in the graph. The features of the defect node may include attributes such as defect type, size, and location, and the features of the edge may represent the spatial adjacency, semantic similarity, and other relationships between defects. In this way, the defect data is converted into a graph structure with attribute information.

[0126] Select appropriate GNN variants, such as graph convolutional networks and graph attention networks, as the basic model for defect severity assessment. The GNN model learns the interaction patterns and contextual information between defects by propagating and aggregating information on node features and edge features. Through multi-layer GNN propagation, node representations containing high-order interaction information can be obtained.

[0127] Specifically, it includes: Graph convolution layer, which is the core component of GCN and is used to aggregate the neighborhood information of nodes on the graph structure. Pooling layer, which is used to reduce the dimension of the graph and extract graph-level feature representation. Common graph pooling operations include maximum pooling, average pooling, and attention pooling. By adopting self-attention pooling and learning the importance weights of nodes, node features are adaptively aggregated to obtain graph-level representation. Classification layer, the classification layer maps graph-level features to the probability distribution of defect categories through a fully connected layer and a softmax function. By optimizing the cross-entropy loss function, GCN can learn the discrimination rules for defect classification.

[0128] After building the GCN model, the model needs to be trained and inferred. During the training process, the defect image with annotations is input into the GCN, the prediction results are calculated through forward propagation, and the model parameters are optimized using the back propagation algorithm. In the inference stage, the defect image to be classified is input into the trained GCN model to obtain the prediction results of the defect category.

[0129] Furthermore, an attention mechanism is introduced in the graph convolution layer to adaptively aggregate node features by learning the importance weights of different neighboring nodes. The attention mechanism can help the model focus on key defect areas and improve classification accuracy. Technologies such as L2 regularization and Dropout are used to control the complexity of the model and prevent overfitting. L2 regularization encourages the model to learn simpler and smoother decision boundaries by adding the L2 norm of the weight to the loss function. Dropout improves the generalization ability of the model by randomly discarding some neurons. At the same time, data enhancement is performed on the defect graph, such as rotation, flipping, scaling, etc., to increase the diversity of training samples. Data enhancement can improve the robustness of the model to defect deformation and noise and reduce the risk of overfitting.

[0130] The annotated defect images and their corresponding graph structure representations are input into the graph convolutional neural network for end-to-end training. The predicted probability distribution of defect categories is calculated through forward propagation, and the cross-entropy loss function is used to evaluate the difference between the predicted results and the true labels. The parameters of the model are then updated through the back-propagation algorithm to optimize the performance of the graph convolutional neural network.

[0131] Use the trained graph convolutional neural network to predict the defect images to be classified, convert the defect images into graph structure representations, input them into the graph convolutional neural network, obtain the predicted probability distribution of the defect categories, and select the category with the highest probability as the final classification result.

[0132] In this embodiment, by modeling the defect area as a graph structure and automatically learning node features and structural information through a graph convolutional neural network, the high-order context and spatial relationship of the defect area can be effectively utilized. An end-to-end process from defect area features to graph structure construction and then to graph convolutional neural network classification is realized. No manual feature design is required, and the optimal feature representation can be automatically learned from the original data. The attention pooling layer can adaptively learn the importance weight of each defect area node, and give greater weight to important nodes when generating graph-level feature representations. The entire graph neural network model can be trained end-to-end, avoiding the inefficiency of staged feature extraction and classification, improving the accuracy and generalization ability of defect classification, and having high training and prediction efficiency.

[0133] Figure 2 FIG. 1 is a schematic diagram of the structure of an intelligent industrial quality inspection system based on AI visual recognition according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0134] The first unit is used to input the collected industrial product image into the pre-trained defect area detection model at the edge, generate multiple candidate defect areas through the region proposal network, and perform defect judgment and bounding box regression on each candidate defect area through the fully connected layer to obtain the defect area detection result;

[0135] The second unit is used to extract defect area features based on the defect area detection results, and perform dimensionality reduction processing to obtain compressed defect area features. According to the adaptive task offloading strategy and the working data of the edge device, it is decided to offload the defect classification task to the cloud or directly perform defect classification on the edge.

[0136] The third unit is used to transmit the compressed defect area features to the cloud through the network if it is decided to offload the defect classification task to the cloud, use the pre-trained defect classification model in the cloud to identify the defect type, output the classification result of the defect area, and return the classification result to the edge end. Combined with the defect area detection result and the defect area classification result, a defect visualization effect diagram and a quality inspection report are generated.

[0137] According to a third aspect of the embodiments of the present invention,

[0138] An electronic device is provided, comprising:

[0139] processor;

[0140] a memory for storing processor-executable instructions;

[0141] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0142] A fourth aspect of the embodiments of the present invention is:

[0143] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0144] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An intelligent industrial quality inspection method based on AI visual recognition, characterized in that: include: At the edge, the collected industrial product images are input into the pre-trained defect area detection model, multiple candidate defect areas are generated through the region proposal network, and defect judgment and bounding box regression are performed on each candidate defect area through the fully connected layer to obtain the defect area detection result; Extract defect area features based on defect area detection results, perform dimensionality reduction processing to obtain compressed defect area features, and decide to offload defect classification tasks to the cloud or perform defect classification directly on the edge based on the adaptive task offloading strategy and the working data of the edge device; If it is decided to offload the defect classification task to the cloud, the compressed defect area features will be transmitted to the cloud through the network, and the pre-trained defect classification model will be used in the cloud to identify the defect type, output the classification results of the defect area, and return the classification results to the edge. Combined with the defect area detection results and the defect area classification results, a defect visualization effect diagram and quality inspection report are generated.

2. The method according to claim 1, characterized in that Before the collected industrial product images are input into the pre-trained defect area detection model at the edge, the following steps are also included: Training the defect area detection model, for each training iteration, randomly selecting a batch of image samples to be trained from the training set, randomly perturbing the pixel value of each image sample to be trained, generating an initial adversarial sample, and iteratively optimizing the initial adversarial sample; In each iteration step, the gradient of the adversarial loss function of the current adversarial sample on the defect area detection model is calculated and normalized to unit length. The current perturbation is updated according to the normalized gradient, and the updated perturbation is added to the original image sample to be trained to obtain a new adversarial sample. The iteration is repeated until the preset conditions are met and the target adversarial sample is output. The output target adversarial sample and the original image sample to be trained are input into the defect area detection model for training, and the first loss function of the target adversarial sample and the second loss function of the original image to be trained are calculated respectively, and the first loss function and the second loss function are fused to obtain the comprehensive training loss; According to the comprehensive training loss, the parameters of the defect area detection model are updated using a parameter updating algorithm, and the parameter updating steps of the defect area detection model are repeated until a preset termination condition is met to obtain a trained defect area detection model.

3. The method according to claim 2, characterized in that The calculation formula for updating the parameters of the defect area detection model using the parameter updating algorithm is as follows: Among them, θ * represents the updated parameters, θ represents the initial model parameters, λ represents the learning rate, β1 represents the exponential decay rate of the control first-order moment estimate, m represents the first-order moment estimate, represents the gradient of the comprehensive training loss to the model parameters, represents the attenuation coefficient of the first-order moment estimate in the tth iteration, t represents the current number of iterations, β2 represents the exponential attenuation rate of the second-order moment estimate, v represents the second-order moment estimate, represents the attenuation coefficient of the second-order moment estimate in the tth iteration, and ε represents a constant parameter.

4. The method according to claim 1, characterized in that According to the adaptive task offloading strategy and the working data of the edge device, the decision to offload the defect classification task to the cloud or to perform defect classification directly on the edge includes: The task offloading problem is modeled as a Markov decision process, and a state space, an action space, and a reward function are defined. The state space includes the edge device state, the network state, and the task state. The action space includes two actions: executing the defect classification task at the local edge or offloading the defect classification task to the cloud. The reward function comprehensively considers the task execution time, energy consumption, and network overhead. A deep reinforcement learning algorithm is used to solve the Markov decision process, calculate the learning objectives of the initial value function network, and update the initial value function network parameters by minimizing the task offloading loss. The iteration is repeated until the preset termination condition is met, and finally the optimal offloading strategy is output; When the actual system is running, the current state observation value of the system is obtained in real time and input into the Markov decision process to obtain the optimal offloading strategy. According to the optimal offloading strategy, the defect classification task is executed locally or offloaded to the cloud. The state observation value includes the computing resources and energy status of the edge device, the network status and the characteristics of the task to be processed.

5. The method according to claim 1, characterized in that The method further comprises: If it is decided to perform defect classification at the edge, a lightweight convolutional neural network is designed at the edge as the first classification model, and a defect dataset is constructed based on the compressed defect area features. The defect classification model deployed in the cloud is used as the second classification model. Using the second classification model to infer the defect data set, obtain soft labels for each sample data on different defect categories, wherein the soft labels include prediction confidence information of the second classification model for the sample category; The soft labels of the second classification model are used as additional supervisory signals to train the first classification model. The comprehensive classification loss function of the first classification model is obtained by calculating the cross entropy loss between the prediction results of the first classification model and the true value, and the divergence between the prediction results of the first classification model and the soft labels of the second classification model. The final trained first classification model is obtained by minimizing the comprehensive classification loss function. The compressed defect area features are input into the trained first classification model, and the classification results of the defect area are output.

6. The method according to claim 5, characterized in that The calculation formula of the comprehensive classification loss function is as follows: Where L represents the comprehensive classification loss function, N represents the sample set, i represents the sample index, α represents the weight coefficient for balancing the true loss and the soft label loss, CE(·) represents the cross entropy loss between the prediction result of the first classification model and the true value, and y i represents the true category label corresponding to the i-th sample, represents the predicted probability of the first classification model for the i-th sample, S represents the first classification model, τ represents the parameter that controls the smoothness of the soft label, KL(·) represents the divergence between the prediction result of the first classification model and the soft label of the second classification model, represents the soft label of the second classification model of the i-th sample, and T represents the second classification model.

7. The method according to claim 1, characterized in that The compressed defect area features are transmitted to the cloud through the network, and the pre-trained defect classification model is used in the cloud to identify the defect type. The classification results of the defect area are output including: The defective regions are regarded as nodes in the graph structure. The features of the nodes include the image attributes of the defective regions. The spatial adjacency between the defective regions is regarded as the edge. The weight of the edge reflects the similarity between the defective regions to construct the graph structure. Construct a graph convolutional neural network, including a graph convolution layer, a pooling layer, and a classification layer. The graph convolution layer receives the node feature matrix and the adjacency matrix of the graph output by the previous layer, and updates the feature representation of the node by aggregating the neighborhood information of each node and applying the convolution operation. The attention pooling layer adaptively aggregates the node features output by the graph convolution layer by learning the importance weight of each node to obtain the feature representation at the graph level. The classification layer inputs the graph-level feature representation output by the attention pooling layer into the fully connected neural network to generate the probability distribution of the defect category. The compressed defect area features are converted into a graph structure representation and input into the trained graph convolutional neural network to obtain the predicted probability distribution of the defect category, and the highest category in the predicted probability distribution is selected as the final classification result.

8. An intelligent industrial quality inspection system based on AI visual recognition, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to input the collected industrial product image into the pre-trained defect area detection model at the edge, generate multiple candidate defect areas through the region proposal network, and perform defect judgment and bounding box regression on each candidate defect area through the fully connected layer to obtain the defect area detection result; The second unit is used to extract defect area features based on the defect area detection results, and perform dimensionality reduction processing to obtain compressed defect area features. According to the adaptive task offloading strategy and the working data of the edge device, it is decided to offload the defect classification task to the cloud or directly perform defect classification on the edge. The third unit is used to transmit the compressed defect area features to the cloud through the network if it is decided to offload the defect classification task to the cloud, use the pre-trained defect classification model in the cloud to identify the defect type, output the classification result of the defect area, and return the classification result to the edge end. Combined with the defect area detection result and the defect area classification result, a defect visualization effect diagram and a quality inspection report are generated.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images