A lightweight neural network model construction method

Abstract

The present application relates to the technical field of neural network model construction, and especially discloses a lightweight neural network model construction method, based on a target task data set, in a lightweight basic operator library containing a depth separable convolution, an inverted residual structure and an attention mechanism module, network structure parameters and weight parameters are jointly optimized through a differentiable neural architecture search to obtain an initial lightweight network model. The initial model is deployed in a target hardware simulation environment, and model precision, inference delay and memory occupation are used as cooperative optimization targets, and a Pareto optimal model cluster is generated through iterative optimization of structure reparameterization and hardware-aware progressive channel pruning. A benchmark student model is selected from the cluster according to deployment constraints, a distillation framework with the initial model as a teacher model is constructed, a mixed strategy is adopted to fine-tune by fusing multi-dimensional distillation loss, and a final model is obtained. The model takes into account the precision and efficiency, and improves the detection efficiency and quality control level.

Description

Technical Field

[0001] This invention relates to the field of neural network model construction technology, specifically a lightweight neural network model construction method. Background Technology

[0002] Existing methods for building lightweight neural network models have several shortcomings. Some methods rely on manual design of the network structure, which not only requires developers to have deep professional knowledge but also has a long design cycle and is difficult to adapt to different target tasks and hardware environments. Some methods only optimize model accuracy or efficiency without considering the co-creation of accuracy, inference latency, and memory usage, resulting in models that are either insufficiently lightweight or suffer from severe accuracy loss.

[0003] Meanwhile, most existing methods do not fully incorporate the characteristics of the target hardware for targeted optimization, which can easily lead to performance mismatch issues after model deployment, failing to fully utilize the hardware's computing power. Furthermore, during the lightweighting process, some methods lack effective feature transfer and accuracy compensation mechanisms, resulting in a decline in the model's ability to extract complex defect features and poor generalization performance, making it difficult to meet the practical application requirements for real-time identification and classification of defects in screen printing images.

[0004] Therefore, developing a method for automatically constructing lightweight neural network models that balances accuracy and efficiency and is compatible with edge devices has become a pressing technical problem in the field of industrial inspection. Summary of the Invention

[0005] The purpose of this invention is to provide a lightweight neural network model construction method to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A lightweight neural network model construction method includes the following steps:

[0008] Step S1: Based on the dataset of the target task, search for differentiable neural architectures in the preset lightweight basic operator library. The lightweight basic operator library includes at least depthwise separable convolution, inverted residual structure and attention mechanism modules. By jointly optimizing the network structure parameters and weight parameters, the initial lightweight network model is obtained.

[0009] Step S2: Deploy the initial lightweight network model in the target hardware simulation environment. With model accuracy, inference latency and memory usage as the co-optimization objectives, perform iterative optimization of the initial lightweight network model by structural reparameterization and hardware-aware progressive channel pruning to generate a Pareto optimal model cluster that represents the trade-off between accuracy and efficiency.

[0010] Step S3: Based on the preset deployment constraints, automatically select the benchmark student model from the Pareto optimal model cluster; construct a distillation framework with the initial lightweight network model as the teacher model and the benchmark student model as the student model, and fine-tune the student model by adopting a hybrid distillation strategy that integrates output logic distillation loss, intermediate layer feature distillation loss and feature map alignment loss based on attention mechanism to obtain the final lightweight neural network model.

[0011] As a preferred embodiment, step S1 includes:

[0012] Construct a neural network search space based on a lightweight basic operator library, wherein the search space is composed of multiple network layers with optional structures connected sequentially, and each network layer contains a candidate operator set consisting of at least two basic operators from the lightweight basic operator library;

[0013] The search space is trained using the dataset of the target task. By introducing architectural parameters associated with the network structure, the gradient descent method is used to jointly optimize the architectural parameters and the weight parameters of each basic operator.

[0014] After joint optimization, candidate operators with architecture parameter values ​​lower than a preset threshold are removed based on the architecture parameter values ​​of the candidate operator set in each layer. The basic operator with the highest architecture parameter value retained in each layer is determined as the final operator of that layer, thus forming the specific structure of the initial lightweight network model.

[0015] As a preferred option, the lightweight basic operator library specifically includes:

[0016] The depthwise separable convolution module is composed of sequentially connected channel-wise convolution and point-wise convolution;

[0017] The inverted residual structure module sequentially includes a first pointwise convolutional layer, a depthwise separable convolutional layer, a squeezed excitation attention layer, and a second pointwise convolutional layer, and establishes a shortcut connection between the input and the output of the second pointwise convolutional layer.

[0018] The multi-head self-attention mechanism module includes a linear projection layer for calculating queries, keys, and values, as well as a multi-head attention calculation layer and an output projection layer based on scaled dot products.

[0019] The depthwise separable convolution module and the inverted residual structure module constitute candidate operators for local feature extraction in the candidate operator set, while the multi-head self-attention mechanism module constitutes candidate operators for global dependency modeling in the candidate operator set.

[0020] As a preferred embodiment, step S2 specifically includes:

[0021] The initial lightweight network model is deployed in the target hardware simulation environment. At least some of the mergeable network structures in the initial lightweight network model are subjected to structural reparameterization operations. The multi-branch topology in the training phase is merged into a single-branch topology in the inference phase to obtain the first intermediate model.

[0022] Based on the target hardware simulation environment, hardware-aware progressive channel pruning is performed on the first intermediate model: based on the channel importance scores of each convolutional layer or attention layer in the first intermediate model, combined with the layer-by-layer inference latency and memory usage data fed back by the target hardware simulation environment, redundant channels that have the least impact on model accuracy and can effectively reduce inference latency and memory usage are iteratively pruned to obtain the second intermediate model.

[0023] Using model accuracy, inference latency, and memory usage as co-optimization targets, structural reparameterization and hardware-aware progressive channel pruning are repeatedly performed on the second intermediate model for multiple rounds of iterative optimization. After each round of iterative optimization, the accuracy-efficiency performance points of the current model are evaluated and recorded.

[0024] Collect the performance points recorded in all iterations, and select all non-dominated accuracy-efficiency performance points from the set of performance points; the optimized models corresponding to each non-dominated accuracy-efficiency performance point are used to form a Pareto optimal model cluster that represents the accuracy-efficiency trade-off.

[0025] As a preferred approach, a baseline student model is automatically selected from the Pareto optimal model cluster based on preset deployment constraints, specifically including:

[0026] Obtain the measured performance metrics of each model in the Pareto optimal model cluster under the target hardware simulation environment. The measured performance metrics include model accuracy, inference latency, and memory usage.

[0027] The Pareto optimal model cluster is filtered according to preset deployment constraints, including the maximum allowable inference latency threshold, the maximum allowable memory usage threshold, and the minimum allowable model accuracy threshold. From the Pareto optimal model cluster, models that simultaneously meet the maximum allowable inference latency threshold, the maximum allowable memory usage threshold, and the minimum allowable model accuracy threshold are selected to form a candidate model set.

[0028] The comprehensive score of each candidate model in the candidate model set is calculated based on the preset deployment scenario priority configuration. The deployment scenario priority configuration defines the weight coefficients of model accuracy, inference latency and memory usage. The candidate model with the highest comprehensive score is selected as the baseline student model.

[0029] As a preferred approach, a distillation framework is constructed, using an initial lightweight network model as the teacher model and a baseline student model as the student model. A hybrid distillation strategy, incorporating output logic distillation loss, intermediate layer feature distillation loss, and attention-based feature map alignment loss, is employed to fine-tune the student model, resulting in the final lightweight neural network model. Specifically, this includes:

[0030] A distillation framework is constructed, fixing the initial lightweight network model as the teacher model and loading the baseline student model as the student model to be fine-tuned.

[0031] Within the distillation framework, based on the dataset of the target task, the total loss of the hybrid distillation strategy is calculated, where:

[0032] The output logic distillation loss is obtained by calculating the KL divergence between the output probability distributions of the teacher model and the student model after softening by the temperature parameter.

[0033] The intermediate layer feature distillation loss is obtained by calculating the L2 distance between the feature maps output by the teacher model and the student model in the predefined corresponding intermediate network layers.

[0034] The feature map alignment loss based on the attention mechanism is obtained by extracting the attention weight map of a specific layer of the teacher model and guiding the feature map of the corresponding layer of the student model to perform attention-weighted feature alignment in the spatial dimension.

[0035] With the goal of minimizing the total loss, the weight parameters of the student model are backpropagated and the gradient is updated to complete the fine-tuning process. The fine-tuned student model is then used as the final lightweight neural network model.

[0036] As a preferred approach, lightweight neural network models are deployed on edge computing devices for real-time defect identification and classification of input screen printing images to be inspected.

[0037] As can be seen from the technical solution provided by the present invention above, the lightweight neural network model construction method provided by the present invention has the following beneficial effects:

[0038] Improve model building efficiency: By adopting a differentiable neural architecture search combined with a lightweight basic operator library, the network structure selection and parameter joint optimization are completed automatically, eliminating the need for manual design of network topology, greatly shortening the model development cycle and reducing the technical threshold;

[0039] Achieving a precise balance between accuracy and efficiency: With model accuracy, inference latency, and memory usage as co-optimization goals, Pareto's optimal model cluster is generated through structural reparameterization and hardware-aware progressive channel pruning. Adaptive models can be flexibly selected according to deployment requirements, ensuring both the lightweight characteristics of the models and the core performance.

[0040] To ensure the accuracy of the lightweight model: The hybrid distillation strategy, which combines output logic distillation loss, intermediate layer feature distillation loss, and feature map alignment loss based on attention mechanism, can fully transfer the high-quality features and inference logic of the teacher model, effectively make up for the accuracy loss that may occur during the lightweighting process, and improve the model's generalization ability.

[0041] Adapting to the real-time application needs of edge devices: After quantization optimization and hardware perception adjustment, the model has low memory consumption and short inference latency, and can be stably deployed on edge computing devices to realize real-time identification and classification of defects in screen printing images, meet the actual needs of high-speed quality inspection in industrial scenarios, and improve production efficiency and quality control level.

[0042] Enhanced technical versatility and adaptability: The lightweight basic operator library covers modules related to local feature extraction and global dependency modeling. The optimization process adapts to the hardware characteristics of different edge devices and can be flexibly applied to various image recognition and classification tasks based on edge computing, making it widely applicable. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the steps involved in constructing a lightweight neural network model according to the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0045] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific embodiments.

[0046] like Figure 1 As shown, this embodiment of the invention provides a lightweight neural network model construction method, including the following steps:

[0047] Step S1: Based on the dataset of the target task, search for differentiable neural architectures in the preset lightweight basic operator library. The lightweight basic operator library includes at least depthwise separable convolution, inverted residual structure and attention mechanism modules. By jointly optimizing the network structure parameters and weight parameters, the initial lightweight network model is obtained.

[0048] Step S2: Deploy the initial lightweight network model in the target hardware simulation environment. With model accuracy, inference latency and memory usage as the co-optimization objectives, perform iterative optimization of the initial lightweight network model by structural reparameterization and hardware-aware progressive channel pruning to generate a Pareto optimal model cluster that represents the trade-off between accuracy and efficiency.

[0049] Step S3: Based on the preset deployment constraints, automatically select the benchmark student model from the Pareto optimal model cluster; construct a distillation framework with the initial lightweight network model as the teacher model and the benchmark student model as the student model, and fine-tune the student model by adopting a hybrid distillation strategy that integrates output logic distillation loss, intermediate layer feature distillation loss and feature map alignment loss based on attention mechanism to obtain the final lightweight neural network model.

[0050] In this embodiment, the core function of step S1 is to construct a search space based on the dataset of the target task using a pre-defined lightweight basic operator library, and to jointly optimize the architecture parameters and weight parameters through differentiable neural architecture search, ultimately selecting an initial network model that meets the lightweight requirements, providing basic network structure support for subsequent iterative optimization; the detailed steps are as follows:

[0051] Step S1-1: Loading the lightweight basic operator library and constructing the search space:

[0052] First, the pre-built lightweight basic operator library is loaded. This library contains three types of core basic operator modules, and the specific composition of each type of module is as follows:

[0053] Depth-separable convolution module: It consists of channel-wise convolution and point-wise convolution connected in sequence. Channel-wise convolution performs convolution operation on each channel of the input feature map separately, and point-wise convolution performs channel dimension fusion on the output of channel-wise convolution through a 1×1 convolution kernel.

[0054] The inverted residual structure module consists of a first pointwise convolutional layer, a depthwise separable convolutional layer, a squeezed excitation attention layer, and a second pointwise convolutional layer. A shortcut connection is established between the module input and the output of the second pointwise convolutional layer. The first pointwise convolutional layer is used to increase the channel dimension of the feature map, and the second pointwise convolutional layer is used to reduce the channel dimension and return to the original dimension.

[0055] Multi-head self-attention mechanism module: includes a linear projection layer, a multi-head attention calculation layer and an output projection layer. The linear projection layer is used to map the input features into query vectors, key vectors and value vectors respectively. The multi-head attention calculation layer calculates and fuses multiple sets of attention weights based on the scaled dot product method. The output projection layer maps the fused attention features into the final output.

[0056] Based on the aforementioned lightweight basic operator library, a neural network search space is constructed. This search space is formed by sequentially connecting multiple network layers with optional structures. Each network layer is configured with a set of candidate operators, which contains at least two basic operators. Specifically, depthwise separable convolution modules and inverted residual structure modules are used as candidate operators for local feature extraction, and multi-head self-attention mechanism modules are used as candidate operators for global dependency modeling. This ensures that each network layer can flexibly choose between local feature extraction and global dependency modeling capabilities.

[0057] Step S1-2: Preprocessing of the target task dataset and initialization of training configuration:

[0058] Preprocessing operations are performed on the dataset for the target task, including data format standardization, data augmentation, and dataset partitioning. Data format standardization converts all input data into tensor format with uniform dimensions and pixel precision to ensure the compatibility of operator computation. Data augmentation expands the amount of training data through operations such as random flipping, cropping, and brightness adjustment to improve the model's generalization ability. The dataset is divided into training and validation sets according to a preset ratio. The training set is used for parameter optimization, and the validation set is used for performance evaluation.

[0059] Initialize the training-related configuration, including introducing architecture parameters directly related to the network structure. These parameters characterize the selection probability of each candidate operator in each layer of the network, and the initial values ​​of the architecture parameters are randomly set according to a uniform distribution. At the same time, initialize the weight parameters of each basic operator, and set the initial values ​​of the weight parameters using the Xavier initialization method. In addition, configure the optimizer to use gradient descent, set training hyperparameters such as learning rate and number of iterations, and define a task-related loss function to quantify the difference between the model's prediction results and the true labels.

[0060] Step S1-3: Joint optimization training of architecture parameters and weight parameters:

[0061] The preprocessed training set is input into the constructed search space to initiate the joint optimization training process. During training, during the forward propagation of the model, the output of each layer is obtained by weighting and summing the outputs of each operator in the candidate operator set of that layer according to the softmax values ​​of the corresponding architecture parameters, i.e.:

[0062] ,in, For the first The output feature map of the layer network, For the first The number of operators in the candidate operator set of the layer. For the first Layer The architecture parameters corresponding to each candidate operator For the first Layer Operation functions of candidate operators For the first Input feature map of the layer network;

[0063] After calculating the model's predicted loss using the loss function, the gradients of the architecture parameters and the weight parameters of each candidate operator are simultaneously calculated using gradient descent. The two types of parameters are then updated according to the following formula:

[0064] ;

[0065] ;

[0066] in, The preset learning rate, The value of the loss function. For the first Layer Weight parameters of each candidate operator For the loss function with respect to architecture parameters gradient, For the loss function with respect to the weight parameters The gradient;

[0067] The forward propagation, loss calculation, and parameter update process are executed iteratively. After each iteration, the model performance is evaluated using a validation set. When the model performance tends to stabilize or reaches the preset number of iterations, the joint optimization training is stopped.

[0068] Steps S1-4: Candidate operator selection and initial lightweight network model generation:

[0069] After joint optimization training is completed, for each layer of the network in the search space, the architecture parameter values ​​corresponding to all candidate operators of that layer are extracted; a preset threshold for architecture parameters is set, and candidate operators with architecture parameter values ​​lower than the threshold are removed, leaving only candidate operators with architecture parameter values ​​higher than the threshold.

[0070] From the candidate operators retained in each layer, the candidate operator with the highest architecture parameter value is selected as the final operator of that layer, and the network structure of that layer is determined. The final operators of all layers are connected in the order of each layer in the search space, and the weight parameters of each layer operator are integrated to form a complete network structure and parameter configuration. The network corresponding to this configuration is the initial lightweight network model. After the initial model is generated, the structure file and parameter file of the model are automatically saved to provide input for the iterative optimization in step S2.

[0071] In this embodiment, the core function of step S2 is to deploy the initial lightweight network model on the target hardware simulation environment. With model accuracy, inference latency, and memory usage as co-optimization objectives, iterative operations of structural reparameterization and hardware-aware progressive channel pruning are used to screen out non-dominated performance points representing the trade-off between accuracy and efficiency, forming a Pareto optimal model cluster. This provides diverse candidate schemes for the subsequent selection of benchmark student models. The detailed steps are as follows:

[0072] Step S2-1: Initial model deployment and structure reparameterization operation:

[0073] The initial lightweight network model generated in step S1 is fully deployed to the target hardware simulation environment. This environment needs to accurately simulate the hardware characteristics of the target edge computing device, including key parameters such as processor architecture, number of computing units, memory bandwidth, and data read / write rate, to ensure that the performance of the model in the simulation environment is consistent with the actual deployment scenario.

[0074] Structural reparameterization is performed on the multi-branch topology in the initial lightweight network model during the training phase. Multi-branch topologies typically contain parallel convolutional branches or composite branches with shortcut connections. While such structures can improve the model's expressive power during training, they increase computational redundancy and latency during inference. The core of structural reparameterization is to fuse the weights of multiple branches into equivalent weights for a single branch. Specific operations include parameter fusion of convolutional layers and batch normalization layers, and weight stacking of multi-branch convolutional kernels. For example, in a topology containing a main branch and shortcut branches, the main branch is a convolutional layer. Batch Normalization Layer The quick branch is a convolutional layer. Batch Normalization Layer After fusion, the weights of the single-branch convolutional kernels With bias Calculate using the following formula:

[0075] ;

[0076] ;

[0077] in, , These are the weights of the convolutional layers in the main branch and the shortcut branch, respectively. parameters For scaling factor, For standard deviation, For the mean, For bias, parameters , , , Meaning and Consistent , These are the biases of the main branch and the fast branch convolutional layers, respectively;

[0078] Through the above fusion operation, the multi-branch topology of the initial model is transformed into a single-branch topology in the inference stage, eliminating computational redundancy between branches and obtaining the first intermediate model. After generating the first intermediate model, its structural integrity and inference feasibility are verified in the target hardware simulation environment to ensure that there are no topological conflicts or hardware incompatibility issues.

[0079] Step S2-2: Hardware-aware progressive channel pruning and second intermediate model generation:

[0080] Based on the characteristics of the target hardware simulation environment, hardware-aware progressive channel pruning is performed on the first intermediate model. The core is to combine channel importance with hardware performance data to filter and remove redundant channels.

[0081] First, calculate the channel importance score for each convolutional layer or attention layer. For convolutional layers, the channel importance score is calculated based on the L1 norm of the kernel weights corresponding to that channel, as shown in the following formula:

[0082] ,in, For the first Importance rating of each output channel This represents the number of input channels for the convolutional layer. The size of the convolution kernel For the first The output channel corresponds to the first The convolution kernels for each input channel are located at position , The weight value at each point; for the attention layer, the channel importance score is calculated based on the mean of the attention weights, that is, the average attention weight value obtained by the channel is used as its importance score;

[0083] Synchronously collect layer-by-layer inference latency and memory usage data fed back from the target hardware simulation environment; inference latency is the time it takes for a layer to complete one forward computation in the simulation environment, and memory usage is the total amount of hardware memory occupied by the weight parameters and intermediate feature maps of that layer;

[0084] A progressive pruning strategy is formulated by combining channel importance scores with hardware performance data. The initial pruning ratio is set to a low value, typically 10% to 20%, prioritizing the pruning of channels that meet the following two conditions: first, the channel importance score is lower than the average score of all channels in the current layer; second, the sum of the layer-by-layer inference latency and memory usage corresponding to the channel is higher than the average level of the current layer. During the pruning process, for each candidate pruning channel, the model accuracy loss is evaluated on the validation set to ensure that the accuracy drop caused by a single pruning does not exceed a preset threshold, which is typically set to 0.5% to 1%.

[0085] After completing this round of pruning, the model is fine-tuned to recover some of the accuracy loss, resulting in the second intermediate model. The fine-tuning process uses gradient descent, with a fixed learning rate of 1 / 10 of the learning rate in the initial joint optimization phase. The number of iterations is 50 to 100 rounds, and only the weight parameters of the unpruned channels are updated.

[0086] Step S2-3: Multi-round iterative optimization and recording of accuracy, efficiency, and performance points:

[0087] With model accuracy, inference latency and memory usage as the co-optimization objectives, the structural reparameterization operation of step S2-1 and the hardware-aware progressive channel pruning of step S2-2 are repeatedly executed on the second intermediate model to start a multi-round iterative optimization process.

[0088] The specific process of each iteration is as follows: First, perform structural reparameterization on the newly added branch structure of the current model (if the branch is generated during fine-tuning or reconstructed after pruning) and merge it into a single-branch topology; then, recalculate the channel importance score based on the updated model weights, and adjust the pruning ratio and candidate channels in combination with the real-time performance feedback of the hardware simulation environment; after performing the pruning operation, perform a short round of fine-tuning to restore accuracy;

[0089] After each round of iteration optimization, a comprehensive performance evaluation of the current model is performed; model accuracy is calculated using the validation set of the target task, and the evaluation metrics are consistent with those in step S1, such as the accuracy of the classification task and the mean squared error of the regression task; inference latency is obtained by averaging the time after running the model 100 times continuously in the target hardware simulation environment; memory usage is collected by the memory monitoring tool of the hardware simulation environment to collect the maximum memory usage after the model is loaded and during the inference process.

[0090] The model accuracy, inference latency, and memory usage after each iteration are combined into a precision-efficiency performance point, which is recorded and stored in the order of iteration rounds to form a set of performance points. The stopping condition for iterative optimization is that the performance points of three consecutive iterations do not show significant improvement, that is, the accuracy improvement is less than 0.3%, the inference latency reduction is less than 5%, and the memory usage reduction is less than 5%, or the number of iterations reaches the preset upper limit, which is usually 10 to 15 rounds.

[0091] Step S2-4: Screening of non-dominated performance points and generation of Pareto optimal model clusters:

[0092] Collect the precision, efficiency, and performance points recorded in all iteration rounds to construct a complete performance point dataset; perform non-dominated performance point filtering on this dataset, using the Pareto dominance criterion as the filtering basis;

[0093] Define Pareto dominance: for performance points With performance point ,set up The corresponding indicators are , , The corresponding indicators are , , ,in, The higher the numerical value, the better the performance, representing the model's accuracy. This represents inference latency; a lower value indicates better performance. This represents memory usage; a lower value indicates better performance. If the following conditions are met... , , If at least one indicator satisfies the strict inequality, then it is called... Dominate , The dominated performance point;

[0094] Iterate through all performance points and determine whether each performance point is dominated by other performance points; retain all non-dominated performance points that are not dominated by any other performance points, and remove all dominated performance points.

[0095] Extract the optimized model corresponding to each non-dominated performance point. These models are optimal in different dimensions of accuracy and efficiency, and cannot improve the performance of one indicator without sacrificing the performance of another. Together, they constitute the Pareto optimal model cluster that represents the trade-off between accuracy and efficiency. Store the structure file, parameter configuration and corresponding performance indicators of the model cluster in the model library to provide data support for the selection of the benchmark student model in step S3.

[0096] In this embodiment, the core function of step S3 is to accurately select a baseline student model that meets the deployment requirements from the Pareto optimal model cluster. By constructing a teacher-student distillation framework and integrating multi-dimensional distillation loss, the student model is fine-tuned. While maintaining the lightweight characteristics of the model, the accuracy loss is compensated, and finally a lightweight neural network model that balances accuracy and efficiency is obtained. The detailed steps are as follows:

[0097] Step S3-1: Performance measurement and data collection of Pareto optimal model cluster:

[0098] For the Pareto optimal model cluster generated in step S2, each model is deployed to the target hardware simulation environment one by one, and performance indicators are tested. The testing process needs to simulate the data flow and computational load of the actual deployment scenario to ensure the authenticity and reliability of the performance data.

[0099] The measured performance indicators include three core data categories:

[0100] Model accuracy: The evaluation metric corresponding to the target task is adopted. For classification tasks, it is accuracy, and for regression tasks, it is mean squared error. It is calculated by running the model on the test set of the target task. The test set must not overlap with the training set and validation set.

[0101] Inference delay: Record the complete time from receiving input data to outputting prediction results. Run the model 200 times consecutively, remove the maximum and minimum values, and take the average as the final inference delay of the model.

[0102] Memory usage: The maximum memory usage during the model loading and inference processes is collected using resource monitoring tools in the hardware simulation environment, including the total memory used by weight parameters, intermediate feature maps, and temporary computation data.

[0103] The model accuracy, inference latency, and memory usage of each model are associated and stored to form a model performance index mapping table, which provides data support for subsequent screening.

[0104] Step S3-2: Deploy constraint filtering and construct candidate model set:

[0105] Load the preset deployment constraints, which consist of three thresholds, namely the maximum allowable inference latency threshold. Maximum allowed memory usage threshold and the minimum allowable model accuracy threshold The three thresholds are pre-set based on the hardware capabilities of the target edge computing device and the actual application requirements. For example, for the task of identifying defects in screen printing images, It can be set to 95%. Set to 50 milliseconds. Set to 100MB;

[0106] Based on the model performance index mapping table, the optimal Pareto model cluster is screened one by one. The screening rule is that the model must simultaneously satisfy the following:

[0107] Model accuracy ;

[0108] Inference delay ;

[0109] Memory usage ;

[0110] Models that do not meet any of the constraints are removed, and all models that meet the constraints are integrated to form a candidate model set; if the candidate model set is empty, the model is downgraded according to a preset ratio. and Each time, reduce by 10%, or reduce Each time it is lowered Re-filter until the set is not empty;

[0111] Step S3-3: Comprehensive score calculation and selection of benchmark student model:

[0112] Load the preset deployment scenario priority configuration, which defines the weight coefficients of model accuracy, inference latency, and memory usage as follows: , , And satisfy The weighting coefficients are adjusted according to the deployment scenario requirements, such as in high-precision priority scenarios. , , Low latency priority scenarios , ,

[0113] First, the performance metrics of each model in the candidate model set are normalized to eliminate differences in dimensions:

[0114] Model accuracy normalization: Let the maximum accuracy in the candidate set be... The minimum precision is Then the normalized accuracy value of a certain model is:

[0115] ,in, This represents the measured accuracy of the model.

[0116] Inference Delay Normalization: Let the maximum delay in the candidate set be... The minimum delay is Then the delayed normalized value of a certain model is:

[0117] ,in, This represents the measured delay of the model;

[0118] Memory usage normalization: Let the maximum memory usage in the candidate set be... Minimum memory usage is Then the normalized value of memory usage for a certain model is:

[0119] ,in, This represents the actual memory usage of the model.

[0120] Based on the normalized indicators and weight coefficients, the comprehensive score S for each candidate model is calculated:

[0121] ;

[0122] The comprehensive scores of all candidate models are sorted in descending order, and the model with the highest score is selected as the baseline student model. Its structure file and weight parameters are saved to prepare for the construction of the distillation framework.

[0123] Step S3-4: Construction of the teacher-student distillation framework:

[0124] The distillation framework is constructed, which includes two core components: a teacher model and a student model.

[0125] Teacher Model: The initial lightweight network model generated in step S1 is fixed as the teacher model, and all its weight parameters are frozen. It is only used to provide high-quality feature representation and output logic and does not participate in parameter updates.

[0126] Student Model: Load the baseline student model selected in step S3-3 as the student model to be fine-tuned. Its network structure maintains a hierarchical correspondence with the teacher model. That is, the input layer, intermediate layer and output layer of the teacher model are matched one by one with the corresponding layers of the student model to ensure the feasibility of feature distillation.

[0127] Configure the training parameters of the distillation framework, including learning rate, number of iterations, batch size, etc.; set the learning rate to 1 / 20 of the learning rate in the initial joint optimization phase, set the number of iterations to 150 to 200 rounds, and adjust the batch size according to the memory capacity of the target hardware simulation environment to ensure that there is no memory overflow during the training process.

[0128] Step S3-5: Calculation of losses during mixed distillation:

[0129] Based on the training set of the target task, the total loss of the hybrid distillation strategy is calculated in the distillation framework. The total loss is composed of the output logical distillation loss, the intermediate layer feature distillation loss and the feature map alignment loss based on the attention mechanism.

[0130] Output logic distillation loss calculation:

[0131] First, the output logits of the teacher and student models are softened by a temperature parameter; let the temperature parameter be... and The teacher model outputs logits as The student model outputs logits as The softened probability distributions are as follows:

[0132] ;

[0133] ;

[0134] in, The normalization function transforms logits into a probability distribution;

[0135] The output logic distillation loss is obtained by calculating the KL divergence of the probability distributions of the two components:

[0136] ;

[0137] in, The number of categories of the target task. For the teacher model to the first Predicted probability of class For the student model to the first The predicted probability of a class;

[0138] Calculation of characteristic distillation loss in the intermediate layer:

[0139] We select predefined corresponding intermediate network layers for the teacher and student models, and let the feature map output by the corresponding intermediate layer of the teacher model be... The feature map output by the intermediate layer of the student model is Both dimensions remain consistent, namely height. ,width Number of channels ;

[0140] The intermediate layer feature distillation loss is obtained by calculating the L2 distance between the feature maps of the two layers:

[0141] ;

[0142] in, The square of the L2 norm. This represents the total number of elements in the feature map, used to normalize the loss value.

[0143] Calculation of feature map alignment loss based on attention mechanism:

[0144] Extracting attention weights from a specific attention layer of the teacher model , Corresponding layer feature map Same dimension, that is Each element represents the attention weight value at the corresponding position; guiding the student model's feature map for the corresponding layer. Attention weighting is applied in the spatial dimension, and the alignment loss between the weighted feature map and the teacher model's weighted feature map is calculated:

[0145] ;

[0146] in, This is an element-wise multiplication operation, which means that the attention weight map is multiplied by each corresponding element of the feature map to achieve spatial dimension attention weighting;

[0147] Total loss fusion:

[0148] The total loss of mixed distillation is obtained by weighting and summing the three types of losses according to preset weighting coefficients. :

[0149] ;

[0150] in, , , For the loss weight coefficients, satisfying The default configuration is , , It can be adjusted according to task requirements;

[0151] Steps S3-6: Student model fine-tuning and final model generation:

[0152] To minimize total loss To achieve this, backpropagation and gradient updates are performed on the weight parameters of the student model; during fine-tuning, only the weight parameters of the student model are updated, while the parameters of the teacher model remain fixed.

[0153] The gradient descent method is used for parameter updates, and the parameter update formula is as follows:

[0154] ;

[0155] in, For the weight parameters of the student model, The learning rate during the fine-tuning phase. For the total loss The gradient;

[0156] After each iteration, the performance of the student model is evaluated using the validation set. If the accuracy improvement on the validation set is less than 0.1% for 20 consecutive iterations, the fine-tuning is considered to have converged and the iteration is stopped. If the convergence condition is not met, the iteration continues until the preset maximum number of iterations is reached.

[0157] After fine-tuning, save the final student model structure file and weight parameters. This model is the final lightweight neural network model that balances accuracy, inference latency, and memory usage. Deploy the model to the target edge computing device, and it can be directly used for real-time defect recognition and classification tasks of inputting screen printing images to be detected.

[0158] When a lightweight neural network model constructed using a lightweight neural network model construction method is deployed on an edge computing device, it is used for real-time defect identification and classification of input screen printing images to be inspected. Through a closed-loop process of image acquisition, preprocessing, real-time inference, and result output, it achieves rapid identification and accurate classification of defects in the screen printing images to be inspected, meeting the needs of real-time quality inspection in industrial scenarios. The detailed implementation process is as follows:

[0159] 1. Edge computing device adaptation and model deployment preparation:

[0160] Edge computing devices need to have low power consumption, small size, and high real-time performance. Their core configuration includes an embedded processor, local memory, a high-speed storage module, and an image acquisition interface. The embedded processor should preferably be a model that supports hardware acceleration, such as the ARM Cortex-A series or a dedicated neural network processor. The local memory capacity should be no less than 2GB to ensure the memory requirements during model loading and inference. The high-speed storage module is used to store model files, preprocessing parameters, and inference logs. The image acquisition interface supports USB 3.0 or Gigabit Ethernet protocol to ensure high-speed transmission of image data.

[0161] Before deploying the model, adaptation and optimization are required: the structure file and weight parameters of the final lightweight neural network model are converted into a format compatible with edge devices, preferably using ONNX or TensorRT engine formats. TensorRT format requires quantization optimization based on the processor architecture of the edge device, converting the model weights from 32-bit floating-point to 16-bit floating-point or 8-bit integer, reducing memory usage and inference latency without significantly sacrificing accuracy. The optimized model file is stored in the high-speed storage module of the edge device, and the model loading parameters are configured, including input image size, pixel value range, inference batch size, etc. The batch size is fixed at 1 to ensure real-time performance.

[0162] 2. Image acquisition and input of the screen printing to be inspected:

[0163] The edge computing device acquires images of the screen printing to be inspected using an industrial camera or image scanner connected to it. The resolution of the acquisition device is no less than 1920×1080 pixels and the frame rate is no less than 30 frames / second, ensuring that the entire area of ​​the screen printing product is covered without motion blur. During the acquisition process, the image data is transmitted to the local memory of the edge device in real time through a preset interface, and the transmission delay is controlled within 10ms.

[0164] The input screen printing images to be detected are uniformly in RGB three-channel image format, supporting two common formats: JPEG and PNG. The aspect ratio of the image is consistent with the aspect ratio of the model input size. If there is a difference, it is adapted by adjusting the parameters of the acquisition device or in the subsequent preprocessing stage to avoid distortion of defect features caused by image stretching.

[0165] 3. Image preprocessing:

[0166] Real-time preprocessing is performed on the input screen printing image to ensure that the image data meets the input requirements of the model. The preprocessing process includes size normalization, pixel value standardization, and noise removal.

[0167] Size normalization: The acquired original image is scaled to the model's preset input size. Let the width of the original image be... Height is The width of the model input dimension is Height is Scaling is performed using bilinear interpolation, and the scaling formula is:

[0168] ;

[0169] ;

[0170] in, , To scale the image in coordinates The width and height are related pixel values ​​at that location. , The coordinates of the original image in the interpolation neighborhood Pixel value at that location, , These are the coordinates of the scaled image;

[0171] Pixel value normalization: This involves converting the normalized image pixel values ​​to the range used during model training. Let the pixel values ​​of the three channels of the image be... , , The standardized formula is:

[0172] ;

[0173] ;

[0174] ;

[0175] in, These are the pixel mean values ​​of the three channels of the training set images, respectively. , , These are the pixel standard deviations of the three channels of the training set images. , , These are the standardized pixel values;

[0176] Noise Removal: Gaussian filtering is used to reduce noise in the standardized image. The filter kernel size is set to 3×3, and the Gaussian kernel function formula is:

[0177] ;

[0178] in, For the Gaussian kernel in coordinates The weight value at that location, The standard deviation of the Gaussian kernel is 0.8. Random noise in industrial environments is eliminated by convolving each pixel in the image with the Gaussian kernel.

[0179] The total time for preprocessing operations is controlled within 5ms to ensure that overall real-time performance is not affected.

[0180] 4. Real-time reasoning and defect feature recognition:

[0181] The edge computing device loads an optimized lightweight neural network model and inputs preprocessed image data into the model for real-time inference. During inference, the model extracts image features through core modules in the lightweight basic operator library: the depthwise separable convolution module and the inverted residual structure module extract local defect features in the image, such as the point features of pinholes and the linear features of scratches; the multi-head self-attention mechanism module models global dependencies in the image, such as the overall features of large-area missing printing areas.

[0182] The forward inference process of the model must meet the real-time requirements, with an inference delay of no more than 50ms for a single image. This delay includes the entire process time of feature extraction, feature fusion, and defect response calculation. During the inference process, the model generates a defect feature response map through the calculation of each layer of the network. Areas with pixel values ​​higher than the preset response threshold in the response map are identified as suspected defect areas. The response threshold is determined based on the statistical analysis of defect samples in the training set and is set to 0.7.

[0183] Coordinate mapping is performed on suspected defect areas to convert the coordinates of suspected defects in the response image into the actual coordinates of the original image. The mapping formula is as follows:

[0184] ;

[0185] ;

[0186] in, , These are the actual coordinates in the original image. , These are the coordinates of suspected defects in the defect response map. , The width and height of the original image. , Input the width and height of the image into the model;

[0187] 5. Defect Classification and Confidence Assessment:

[0188] Based on the extracted defect features, the model calculates the probability distribution of each type of defect using the Softmax function in the output layer to achieve defect classification; let the total number of defect categories be... The model outputs logits as , … The formulas for calculating the probability of each category are as follows:

[0189] ;

[0190] in, For the first Predicted probability of class defects For the first The logits value corresponding to the class defect. The number of defect categories includes five common screen printing defects: pinholes, scratches, omissions, deformation, and contamination.

[0191] With a confidence threshold set at 0.9, the predicted probability of a certain type of defect... If the probability exceeds the threshold, the image to be detected is determined to contain the defect of that type; if the predicted probability of all categories is below the threshold, the image to be detected is determined to be defect-free; at the same time, the area and morphological parameters of the defect region are calculated. The area is obtained by counting the number of pixels in the suspected defect region in the original image and converting it into the actual physical area. The morphological parameters are obtained by calculating the roundness, aspect ratio and other indicators of the defect region, providing supplementary information for subsequent quality inspection decisions.

[0192] 6. Results Output and Log Recording:

[0193] After defect identification and classification are completed, the edge computing device outputs the results in multiple ways: the local LCD screen displays the defect type, defect location coordinates, confidence value and processing time in real time; the serial port interface transmits the results to the industrial control equipment to trigger subsequent sorting or marking actions; the network interface supports uploading the results to the local server to realize centralized management of quality inspection data.

[0194] Meanwhile, the edge device automatically records log information for each detection, including detection time, image file name, defect type, confidence level, processing time, device status, etc. The log files are stored by date and retained for no less than 30 days to facilitate subsequent quality traceability and model optimization.

[0195] The entire testing process takes no more than 60ms, meeting the real-time quality inspection needs in industrial scenarios. The model's defect identification accuracy is no less than 95%, and its classification accuracy is no less than 92%, ensuring the reliability of the test results.

[0196] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight neural network model construction method, characterized in that: Includes the following steps: Step S1: Based on the dataset of the target task, search for differentiable neural architectures in a pre-defined lightweight basic operator library. The lightweight basic operator library includes at least depthwise separable convolution, inverted residual structure, and attention mechanism modules. By jointly optimizing the network structure parameters and weight parameters, an initial lightweight network model is obtained. The lightweight basic operator library specifically includes: The depthwise separable convolution module is composed of sequentially connected channel-wise convolution and point-wise convolution; The inverted residual structure module sequentially includes a first pointwise convolutional layer, a depthwise separable convolutional layer, a squeezed excitation attention layer, and a second pointwise convolutional layer, and establishes a shortcut connection between the input and the output of the second pointwise convolutional layer. The multi-head self-attention mechanism module includes a linear projection layer for calculating queries, keys, and values, as well as a multi-head attention calculation layer and an output projection layer based on scaled dot products. The depthwise separable convolution module and the inverted residual structure module constitute candidate operators for local feature extraction in the candidate operator set, and the multi-head self-attention mechanism module constitutes candidate operators for global dependency modeling in the candidate operator set. Step S2: Deploy the initial lightweight network model in the target hardware simulation environment, perform structural reparameterization operation on at least some mergeable network structures in the initial lightweight network model, merge the multi-branch topology of the training phase into the single-branch topology of the inference phase, and obtain the first intermediate model. Based on the target hardware simulation environment, hardware-aware progressive channel pruning is performed on the first intermediate model: based on the channel importance scores of each convolutional layer or attention layer in the first intermediate model, combined with the layer-by-layer inference latency and memory usage data fed back by the target hardware simulation environment, redundant channels that have the least impact on model accuracy and can effectively reduce inference latency and memory usage are iteratively pruned to obtain the second intermediate model. Using model accuracy, inference latency, and memory usage as co-optimization targets, structural reparameterization and hardware-aware progressive channel pruning are repeatedly performed on the second intermediate model for multiple rounds of iterative optimization. After each round of iterative optimization, the accuracy-efficiency performance points of the current model are evaluated and recorded. Collect the performance points recorded in all iterations, and select all non-dominated accuracy-efficiency performance points from the set of performance points; combine the optimized models corresponding to each non-dominated accuracy-efficiency performance point to form a Pareto optimal model cluster that represents the accuracy-efficiency trade-off. Step S3: Based on the preset deployment constraints, automatically select the baseline student model from the Pareto optimal model cluster; A distillation framework is constructed, fixing the initial lightweight network model as the teacher model and loading the baseline student model as the student model to be fine-tuned. Within the distillation framework, based on the dataset of the target task, the total loss of the hybrid distillation strategy is calculated, where: The output logic distillation loss is obtained by calculating the KL divergence between the output probability distributions of the teacher model and the student model after softening by the temperature parameter. The intermediate layer feature distillation loss is obtained by calculating the L2 distance between the feature maps output by the teacher model and the student model in the predefined corresponding intermediate network layers. The feature map alignment loss based on the attention mechanism is obtained by extracting the attention weight map of a specific layer of the teacher model and guiding the feature map of the corresponding layer of the student model to perform attention-weighted feature alignment in the spatial dimension. With the goal of minimizing the total loss, the weight parameters of the student model are backpropagated and the gradient is updated to complete the fine-tuning process. The fine-tuned student model is then used as the final lightweight neural network model.

2. The lightweight neural network model construction method according to claim 1, characterized in that: Step S1 includes: Construct a neural network search space based on a lightweight basic operator library, wherein the search space is composed of multiple network layers with optional structures connected sequentially, and each network layer contains a candidate operator set consisting of at least two basic operators from the lightweight basic operator library; The search space is trained using the dataset of the target task. By introducing architectural parameters associated with the network structure, the gradient descent method is used to jointly optimize the architectural parameters and the weight parameters of each basic operator. After joint optimization, candidate operators with architecture parameter values ​​lower than a preset threshold are removed based on the architecture parameter values ​​of the candidate operator set in each layer. The basic operator with the highest architecture parameter value retained in each layer is determined as the final operator of that layer, thus forming the specific structure of the initial lightweight network model.

3. The lightweight neural network model construction method according to claim 1, characterized in that: The baseline student model is automatically selected from the optimal Pareto model cluster based on preset deployment constraints, specifically including: Obtain the measured performance metrics of each model in the Pareto optimal model cluster under the target hardware simulation environment. The measured performance metrics include model accuracy, inference latency, and memory usage. The Pareto optimal model cluster is filtered according to preset deployment constraints, including the maximum allowable inference latency threshold, the maximum allowable memory usage threshold, and the minimum allowable model accuracy threshold. From the Pareto optimal model cluster, models that simultaneously meet the maximum allowable inference latency threshold, the maximum allowable memory usage threshold, and the minimum allowable model accuracy threshold are selected to form a candidate model set. The comprehensive score of each candidate model in the candidate model set is calculated based on the preset deployment scenario priority configuration. The deployment scenario priority configuration defines the weight coefficients of model accuracy, inference latency and memory usage. The candidate model with the highest comprehensive score is selected as the baseline student model.

4. The lightweight neural network model construction method according to claim 1, characterized in that: The lightweight neural network model is deployed on an edge computing device for real-time defect identification and classification of input screen printing images to be inspected.

Images