Knowledge distillation-based domain large model lightweight adaptive method and system

Abstract

The application relates to the technical field of large model algorithm, in particular to a domain large model lightweight adaptive method and system based on knowledge distillation, which obtains knowledge distillation parameters and student model performance parameters, establishes a nonlinear mapping relationship between the two, optimizes and solves the best parameter combination based on the mapping relationship, generates target knowledge distillation parameters, and issues the target parameters to a training engine, monitors performance deviation in real time and triggers re-optimization, monitors performance fluctuation and controls model characteristics in the inference process, collects target domain data characteristics, combines big data analysis to correct the mapping relationship, and improves domain adaptation capability, builds a knowledge base and a case base of historical distillation data, forms a standardized adjustment scheme, and realizes adaptive matching. Through accurate modeling, dynamic optimization, real-time monitoring and knowledge reuse, the scheme significantly improves the knowledge distillation efficiency, model robustness and domain adaptability, and provides systematic technical support for large model lightweight.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model algorithm, in particular to a domain large model lightweight adaptive method and system based on knowledge distillation. BACKGROUND

[0002] With the in-depth application of artificial intelligence technology in various fields, domain large models have become a core technology support due to their powerful semantic understanding and task processing capabilities. However, large model parameters are large in size and high in computational complexity, and in edge device deployment, real-time inference and other scenarios, they face problems such as bloated model size, low inference efficiency, high energy consumption, and difficulty in meeting lightweight application requirements. Knowledge distillation, as a mainstream model lightweight technology, can transfer the knowledge of a large model (teacher model) to a lightweight model (student model), while preserving core performance and compressing model size. However, existing methods generally have problems such as rough parameter adjustment and insufficient adaptive ability.

[0003] The above content is only used to assist in understanding the technical solutions of the present application and does not represent the acknowledgement of the above content as prior art. SUMMARY

[0004] The main purpose of the present application is to provide a domain large model lightweight adaptive method and system based on knowledge distillation, aiming to solve the technical problems of rough parameter adjustment and insufficient adaptive ability in transferring the knowledge of a traditional large model to a lightweight model.

[0005] To achieve the above purpose, the present application provides a domain large model lightweight adaptive method based on knowledge distillation, which comprises:

[0006] Obtain the knowledge distillation parameters of the domain large model and the student model performance parameters, and establish a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters;

[0007] According to the constructed nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters, the optimal combination of knowledge distillation parameters is optimized and solved, and the target knowledge distillation parameters that achieve the target performance indicators are obtained;

[0008] The target knowledge distillation parameters obtained by optimization are sent to the model training engine, the student model performance parameters are collected in real time and the process is monitored, and re-optimization is triggered when the student model performance parameters deviate;

[0009] Monitor the student model performance parameters during model inference, and control the model characteristics in scenes where the student model performance parameters fluctuate irregularly or uncontrollably;

[0010] Obtaining target domain data characteristic data, performing big data analysis on student model performance parameters, knowledge distillation parameters and target domain data characteristic data of a training process, and continuously correcting and improving the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters according to the internal relationship between the parameters;

[0011] Obtaining historical distillation data under different target domain data characteristics, the historical distillation data including a historical target knowledge distillation parameter combination, establishing a knowledge distillation parameter optimization knowledge base and case base, forming a standardized knowledge distillation lightweight scheme, and performing self-adaptive matching and adjustment on the knowledge distillation parameters of the domain large model based on the standardized knowledge distillation lightweight scheme.

[0012] Optionally, the knowledge distillation parameters include a teacher model temperature coefficient, a knowledge weight coefficient and a feature alignment loss weight, and the student model performance parameters include a model accuracy and a reasoning delay, the obtaining of the knowledge distillation parameters and the student model performance parameters of the domain large model and the establishment of the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters include:

[0013] Collecting real-time data of the teacher model temperature coefficient, the knowledge weight coefficient and the feature alignment loss weight, measuring indexes of the model accuracy and the reasoning delay, and performing normalization processing on the collected original data;

[0014] Removing noise in the data after normalization processing by using a moving average filtering method, and identifying abnormal values by using a box plot method;

[0015] Based on the filtered data, a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is established by using a support vector machine algorithm.

[0016] Optionally, the knowledge distillation parameters include a teacher model temperature coefficient, a knowledge weight coefficient and a feature alignment loss weight, and the student model performance parameters include a model accuracy and a reasoning delay, the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is constructed, and a Bayesian optimization algorithm is used for intelligent optimization, wherein the optimization steps include:

[0017] According to a preset parameter coding mode, the three parameters of the teacher model temperature coefficient, the knowledge weight coefficient and the feature alignment loss weight are coded into a real number vector, and the vector dimension is set according to the parameter accuracy requirement;

[0018] Obtaining the set real number vector, setting a collection function of the Bayesian optimization algorithm, and converting the model accuracy and the reasoning delay into dimensionless indexes by using a normalization processing method;

[0019] After the acquisition function is set, a Gaussian process regression model is used to construct the surrogate function, and the exploration-exploitation balance factor of the Bayesian optimization algorithm is set to a preset value, and the parameter iterative update is performed in a sequential optimization manner;

[0020] According to the iterative process of the Bayesian optimization algorithm, the number of initial sampling points is set to a preset value, and the maximum number of iterations is set to a preset value;

[0021] When the Bayesian optimization algorithm reaches the convergence condition or the maximum number of iterations, the position vector corresponding to the optimal parameter is obtained by decoding, and is used as the target knowledge distillation parameter of the target performance indicator.

[0022] Optionally, the target knowledge distillation parameter obtained by optimization is sent to the model training engine, and the student model performance parameter is collected in real time and process monitoring is performed, and re-optimization is triggered when the student model performance parameter deviates, comprising:

[0023] The target knowledge distillation parameter obtained by optimization is obtained, and the target knowledge distillation parameter is sent to the automatic adjustment module of the model training engine through a preset communication interface;

[0024] According to the target knowledge distillation parameter, the teacher model temperature coefficient, the knowledge weight coefficient and the feature alignment loss weight are controlled, and the teacher model temperature coefficient, the knowledge weight coefficient and the feature alignment loss weight are monitored in real time;

[0025] If the actual parameter monitored deviates from the target knowledge distillation parameter, an adaptive sliding mode control algorithm is used to dynamically adjust the input signal of the training hyperparameter;

[0026] The performance monitor is used to continuously collect the model accuracy and reasoning delay, and the particle filtering algorithm is used to obtain the optimal estimated student model performance parameter;

[0027] The trend of the optimal estimated student model performance parameter is calculated using the exponential weighted moving average method, and the control chart method is used for anomaly detection;

[0028] If the model accuracy or reasoning delay of three consecutive samplings exceeds the preset threshold of the target value, the knowledge distillation parameter re-optimization program is triggered, and the optimal combination of the knowledge distillation parameter is recalculated.

[0029] Optionally, the model characteristics include model generalization ability and reasoning stability, the student model performance parameter is monitored during model reasoning, and the model characteristics are controlled in the scene where the student model performance parameter fluctuates irregularly or uncontrollably, comprising:

[0030] The statistical process control chart is used to monitor the change trend of the student model performance parameter, and the statistical process control chart applies the eight discrimination criteria of the Sigma control chart;

[0031] determining whether the eight differentiating criteria meet the differentiating condition;

[0032] If the differentiating condition is not met, it is determined that the student model performance parameters do not have irregular fluctuations or are uncontrollable, and the subsequent reasoning task is continued to be executed;

[0033] If the differentiating condition is met, it is determined that the student model performance parameters have irregular fluctuations or are uncontrollable, triggering a detection and analysis program for model characteristics, and feeding back the detection and analysis results to the model reasoning engine for adaptive compensation.

[0034] Optionally, the target domain data characteristic data includes data distribution offset degree, domain semantic complexity and user task adaptation degree indicators, the target domain data characteristic data is obtained, big data analysis is performed on the student model performance parameters, the knowledge distillation parameters and the target domain data characteristic data in the training process, and the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is constantly corrected and improved according to the internal relationship between the parameters, including:

[0035] Obtain the distribution offset degree data, the semantic complexity data and the user task adaptation degree indicators of the target domain data, and obtain the knowledge distillation parameters and the student model performance parameters from the training system;

[0036] Perform outlier detection on the distribution offset degree data, the semantic complexity data, the user task adaptation degree indicators, the knowledge distillation parameters and the student model performance parameters, identify outliers using the box plot method, and perform smoothing processing on data exceeding the preset multiple quartile range;

[0037] According to the smoothed data, perform dimensionality reduction using principal component analysis, and extract main characteristic variables;

[0038] Using the main characteristic variables, perform association rule analysis using the FP-growth algorithm, mine the association rules between the knowledge distillation parameters, the student model performance parameters and the target domain data characteristics, and obtain the rule that if the temperature coefficient is in a specified range and the knowledge weight coefficient is in a specified range, the user task adaptation degree score is above a specified value;

[0039] Adaptive response surface method is used to correct the model, new sampling points are generated by central composite experimental design, and the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is fitted by using the weighted least squares method.

[0040] Optionally, the method further comprises: acquiring historical distillation data under different target field data characteristics, the historical distillation data comprising historical target knowledge distillation parameter combinations; establishing a knowledge base and a case base of knowledge distillation parameter optimization; forming a standardized knowledge distillation lightweight scheme; and performing self-adaptive matching adjustment on the knowledge distillation parameters of the field large model based on the standardized knowledge distillation lightweight scheme.

[0041] Acquiring historical distillation data under different target field data characteristics, the historical distillation data comprising historical target knowledge distillation parameter combinations, and performing clustering analysis by using a K-means algorithm;

[0042] Performing outlier detection on the data according to the clustering analysis result, identifying outliers by using a boxplot method, and replacing outliers by using a mean value when data exceeds a preset interquartile range;

[0043] Establishing a rule-based expert knowledge base, the expert knowledge base comprising if-then rules;

[0044] Constructing a case base by using a case-based reasoning algorithm, the case base comprising target field data characteristics, knowledge distillation parameter combinations and actual reasoning effects, and calculating similarity between cases by using a cosine similarity;

[0045] Receiving target field data characteristics input, performing rule-based reasoning according to the expert knowledge base, retrieving similar cases from the case base, obtaining target knowledge distillation parameters by using a fuzzy comprehensive evaluation method, and performing self-adaptive matching adjustment on the field large model based on the target knowledge distillation parameters.

[0046] In addition, to achieve the above object, the application further provides a field large model lightweight self-adaptive system based on knowledge distillation, comprising:

[0047] A parameter modeling module is configured to acquire knowledge distillation parameters of a field large model and student model performance parameters, and establish a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters;

[0048] A parameter optimization module is configured to optimize and solve an optimal combination of knowledge distillation parameters according to the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters, and obtain target knowledge distillation parameters for achieving target performance indicators;

[0049] A monitoring and adjusting module is configured to send the target knowledge distillation parameters obtained by optimization to a model training engine, collect student model performance parameters in real time and perform process monitoring, and trigger re-optimization when the student model performance parameters deviate.

[0050] The characteristic control module is configured to monitor the student model performance parameters in the model inference process, and control the model characteristics in the case of irregular fluctuations or uncontrollable student model performance parameters.

[0051] The model correction module is configured to obtain target field data characteristics data, perform big data analysis on the student model performance parameters, the knowledge distillation parameters and the target field data characteristics data in the training process, and constantly correct and improve the non-linear mapping relationship between the knowledge distillation parameters and the student model performance parameters according to the internal relationship between the parameters.

[0052] The knowledge adaptation module is configured to obtain historical distillation data under different target field data characteristics, the historical distillation data including historical target knowledge distillation parameter combinations, establish a knowledge base and a case base for knowledge distillation parameter optimization, form a standardized knowledge distillation lightweight scheme, and perform self-adaptive matching and adjustment on the knowledge distillation parameters of the field large model based on the standardized knowledge distillation lightweight scheme.

[0053] In addition, to achieve the above-mentioned purposes, the application further provides a field large model lightweight self-adaptive device based on knowledge distillation, which comprises a memory, a processor and a field large model lightweight self-adaptive program based on knowledge distillation stored on the memory and executable on the processor.

[0054] In addition, to achieve the above-mentioned purposes, the application further provides a storage medium, wherein the storage medium stores a field large model lightweight self-adaptive program based on knowledge distillation, and the field large model lightweight self-adaptive program based on knowledge distillation implements the steps of the field large model lightweight self-adaptive method based on knowledge distillation when executed by a processor.

[0055] The application provides a domain large model lightweight adaptive method based on knowledge distillation, which collects knowledge distillation parameters and student model performance parameters, establishes a nonlinear mapping relationship by using algorithms such as support vector machines, breaks the experience dependence of traditional manual parameter adjustment, and realizes quantitative association modeling of parameters and performance. This mechanism can accurately capture the complex influence of parameter combination on model performance, avoid blind trial and error, change the knowledge distillation process from experience-driven to data-driven, significantly improve the distillation efficiency and the targeting of student model performance optimization; by issuing the optimized target parameters to the training engine and monitoring the performance deviation in real time, combining algorithms such as fuzzy PID and Kalman filtering to realize dynamic adjustment and re-optimization, the uncertainty factors such as environmental changes and data distribution fluctuations in the training process can be effectively coped with. When the performance of the student model exceeds the preset threshold, the re-optimization program is automatically triggered to avoid continuous performance degradation and ensure the stability and robustness of the model in dynamic scenarios; in view of the irregular fluctuations that may occur in the model reasoning process, the scheme introduces statistical process control and sigma difference criterion to monitor the performance parameter trend in real time and trigger adaptive compensation of optical properties. This mechanism can identify abnormal scenarios in advance and intervene to avoid reasoning quality decline caused by improper parameter configuration, ensuring the reliability of the model in actual deployment; by collecting the data characteristics of the target domain and combining big data analysis, using principal component analysis, association rule mining and other technologies to correct the mapping relationship between knowledge distillation parameters and performance, the model can dynamically adapt to different domain data characteristics, improve the knowledge transfer relevance, solve the problem of insufficient adaptation to domain differences in traditional methods, and enhance the generalization ability of the model in cross-domain scenarios; by constructing a knowledge base and a case base of historical distillation data, the optimization experience of different domains is structured and deposited, and combined with rule-based reasoning and case retrieval technology, a standardized adjustment scheme is formed. When facing new domain tasks, similar cases can be quickly matched and target parameters can be generated to avoid repeated parameter adjustment, significantly reducing the time and computational cost in engineering application, and improving the landing efficiency of the lightweight solution. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 FIG. 1 is a hardware running environment based on knowledge distillation of a domain large model lightweight adaptive device structure schematic diagram involved in an embodiment scheme of the application;

[0057] Figure 2 FIG. 2 is a flowchart of an embodiment of the application of the domain large model lightweight adaptive method based on knowledge distillation;

[0058] Figure 3 FIG. 3 is a structural block diagram of an embodiment of the application of the domain large model lightweight adaptive system based on knowledge distillation.

[0059] The implementation, functional features and advantages of the application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] Reference Figure 1 , Figure 1 This is a schematic diagram of the structure of a lightweight adaptive device for domain large models based on knowledge distillation, which is part of the hardware operating environment of the embodiment of the present invention.

[0062] like Figure 1 As shown, this lightweight adaptive device for large domain models based on knowledge distillation may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, and optionally, it may also include a standard wired interface or a wireless interface. In this invention, the wired interface of the user interface 1003 may be a USB interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed random access memory (RAM) or non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0063] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on lightweight adaptive devices for large domain models based on knowledge distillation, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a lightweight adaptive program for a large domain model based on knowledge distillation.

[0065] exist Figure 1In the knowledge distillation-based lightweight adaptive domain large model device shown, the network interface 1004 is mainly used to connect to the backend server and communicate with the backend server; the user interface 1003 is mainly used to connect to peripheral devices; the knowledge distillation-based lightweight adaptive domain large model device calls the knowledge distillation-based lightweight adaptive program stored in the memory 1005 through the processor 1001 and executes the knowledge distillation-based lightweight adaptive domain large model method provided in this embodiment of the invention.

[0066] Based on the above hardware structure, an embodiment of the present invention, a lightweight adaptive method for domain-wide large models based on knowledge distillation, is proposed.

[0067] Reference Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the lightweight adaptive method for large domain models based on knowledge distillation according to the present invention.

[0068] In one embodiment, the lightweight adaptive method for large domain models based on knowledge distillation includes the following steps:

[0069] Step S100: Obtain the knowledge distillation parameters of the domain large model and the performance parameters of the student model, and establish a nonlinear mapping relationship between the knowledge distillation parameters and the performance parameters of the student model.

[0070] Knowledge distillation parameters refer to the control variables involved in the knowledge transfer process between the teacher and student models, including temperature coefficient, loss function weights, and the ratio of distillation loss to classification loss. These parameters can be obtained through parameter configuration records during the training process of the teacher and student models. For example, the temperature coefficient is used to adjust the smoothness of the probability distribution output by the teacher model, and the loss function weights determine the contribution ratio of different loss terms. Student model performance parameters are quantitative indicators that measure the inference performance of the student model, including accuracy, F1 score, inference latency, model size, and energy consumption. These can be obtained in real time through performance monitoring modules during model training and inference. For example, accuracy reflects the correctness of the classification task, and inference latency reflects the model's response speed. The nonlinear mapping relationship is a mathematical expression describing the complex relationship between the combination of knowledge distillation parameters and the student model performance parameters. It can be constructed using algorithms such as Support Vector Machine (SVM) and neural network regression. For example, SVM uses kernel functions to map the low-dimensional parameter space to a high-dimensional feature space, capturing the implicit impact of parameter combinations on performance. The technical operation involves collecting parameter configurations and performance indicators through a data acquisition module to form a parameter performance dataset. Nonlinear modeling algorithms, such as Gaussian process regression, are then used to train the data to generate a mapping model, thereby replacing the traditional manual trial and error method and reducing the blindness of parameter adjustment.

[0071] Step S200: Based on the nonlinear mapping relationship between the constructed knowledge distillation parameters and the student model performance parameters, optimize the solution of the best combination of knowledge distillation parameters to obtain the target knowledge distillation parameters that achieve the target performance index.

[0072] The optimization process involves using mathematical optimization algorithms to search for the optimal solution that satisfies the constraints in the parameter space. Methods include genetic algorithms, particle swarm optimization, and Bayesian optimization. For example, Bayesian optimization uses a Gaussian process surrogate model to gradually narrow the search range to improve efficiency. The target performance metric is a user-defined multi-objective optimization condition, such as minimizing model volume while maintaining accuracy loss below 5%, or balancing inference latency and energy consumption. This can be generated through user input or preset rules. The technical operation involves using the mapping model in step S100 to transform the target metric into mathematical constraints. An optimization algorithm iteratively generates a candidate set of parameters, calculates their predicted performance values, and selects the optimal solution that satisfies the constraints. This avoids the reliance on experience in traditional parameter tuning and improves the accuracy of parameter combinations.

[0073] Step S300: The optimized target knowledge distillation parameters are sent to the model training engine, the student model performance parameters are collected in real time and the process is monitored, and re-optimization is triggered when the student model performance parameters deviate.

[0074] Real-time monitoring involves continuously tracking changes in the student model's performance parameters during training via a data acquisition module. This includes metrics such as accuracy and loss after each iteration. For example, tools like TensorBoard can be used for visual tracking. Performance deviation refers to the deviation between actual performance and the target metric exceeding a preset threshold. This may be caused by changes in data distribution or fluctuations in hardware resources. For example, an alert is triggered when the accuracy drops by more than 2%. Technical operation involves passing the optimized parameter configuration to the training engine to start model training. The monitoring module continuously collects data and compares it with a threshold. When a deviation exceeds the threshold, a re-optimization process is triggered, re-invoking the nonlinear mapping model and optimization algorithm to generate corrected parameters. For example, fuzzy PID or Kalman filtering algorithms can be used to dynamically adjust parameters to adapt to environmental changes.

[0075] Step S400: Monitor the performance parameters of the student model during the model inference process, and control the model characteristics in scenarios where the performance parameters of the student model fluctuate irregularly or are uncontrollable.

[0076] Model inference monitoring involves real-time recording of the student model's inference results in the deployment environment, including output confidence, inference latency, and energy consumption. For example, this can be achieved by embedding a lightweight monitoring agent into the inference framework. Irregular fluctuations refer to performance parameters exceeding the statistical control range due to abnormal input data or sudden changes in hardware load. For example, inference latency might suddenly increase to 1.5 times the threshold. The technical operation involves constructing a control chart and calculating statistics such as the mean and standard deviation. When a parameter exceeds the 3σ threshold, it is identified as an abnormal fluctuation, triggering an adaptive compensation mechanism. For example, this might involve dynamically adjusting the model quantization bit width or enabling a backup model path to suppress performance degradation.

[0077] Step S500: Obtain target domain data characteristics data, perform big data analysis on student model performance parameters, knowledge distillation parameters and target domain data characteristics data during the training process, and continuously correct and improve the nonlinear mapping relationship between knowledge distillation parameters and student model performance parameters based on the inherent relationship between each parameter.

[0078] The target domain data characteristics refer to features such as data distribution, noise level, and class imbalance in the new domain. Examples include the density of domain terms in the text domain or illumination variations in the image domain. Big data analytics techniques include Principal Component Analysis (PCA) for dimensionality reduction and Association Rule Mining (ARM) to discover dependencies between parameters and data characteristics. For example, ARM can eliminate redundant feature dimensions, and ARM can identify the association patterns between temperature coefficients and illumination variations. The technical operation involves collecting data from the target domain and extracting statistical features. This data is then combined with parameters and performance data from the training process to construct a multi-dimensional dataset. Redundant features are eliminated using PCA, and association rule mining is used to identify dependencies between key parameters and data characteristics. This process updates the parameter weights of the nonlinear mapping model, enhancing the model's adaptability to domain differences.

[0079] Step S600: Obtain historical distillation data under different target domain data characteristics. The historical distillation data includes historical target knowledge distillation parameter combinations. Establish a knowledge base and case base for optimizing knowledge distillation parameters, form a standardized lightweight knowledge distillation scheme, and adaptively match and adjust the knowledge distillation parameters of the domain large model based on the standardized lightweight knowledge distillation scheme.

[0080] The knowledge base and case library are structured databases that store historical distillation task parameter configurations, domain data characteristics, and optimization paths. For example, they might contain records linking temperature coefficient settings and category imbalance data in a specific medical imaging domain. The technical operation involves storing historical distillation data in a structured manner. When facing a new domain, its data characteristics are first extracted, and similar historical cases are found through case retrieval, such as cosine similarity matching. This is combined with a rule engine to generate initial parameter suggestions, which are then further optimized through steps S100 to S500, thereby reducing redundant parameter tuning and improving cross-domain deployment efficiency.

[0081] This embodiment provides a lightweight adaptive method for domain-wide large-scale models based on knowledge distillation. It quantifies the impact of parameters by establishing a nonlinear mapping relationship between knowledge distillation parameters and performance parameters. An optimization algorithm searches for the optimal parameter combination to meet target indicators. Combined with real-time monitoring and re-optimization mechanisms, parameters are dynamically adjusted to cope with environmental changes. Statistical process control and adaptive compensation techniques are employed to suppress abnormal fluctuations during the inference stage. Through big data analysis and association with domain data characteristics, the mapping model is continuously updated to enhance cross-domain adaptability. Finally, a structured knowledge base is formed to achieve standardized solution reuse. This method can achieve the following technical effects: reducing manual trial-and-error costs, improving the accuracy of parameter combinations, rapidly responding to environmental changes, maintaining model performance stability, reducing deployment time in new domains, enhancing model generalization, intervening in abnormal scenarios in advance, and accelerating the deployment of multi-domain models.

[0082] In one embodiment, the knowledge distillation parameters include the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight; the student model performance parameters include model accuracy and inference latency. The knowledge distillation parameters of the domain-wide model and the student model performance parameters are obtained, and a non-linear mapping relationship between the knowledge distillation parameters and the student model performance parameters is established, including:

[0083] Real-time data of teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight were collected, and the model accuracy and inference latency were measured. The collected raw data were then normalized.

[0084] The teacher model temperature coefficient is a parameter used to soften the probability distribution of the teacher model's output. Adjusting the temperature value (e.g., increasing from 1 to 10) can reduce the confidence of class predictions. For example, its value ranges from 0.1 to 10. The knowledge weight coefficient controls the proportion of knowledge distillation loss to traditional classification loss in the total loss function. For example, its value ranges from 0 to 1. The feature alignment loss weight adjusts the similarity loss weight of the intermediate feature maps between the teacher and student models. For example, its value ranges from 0 to 1. Model accuracy is the proportion of correctly predicted samples in a classification task, which can be obtained through evaluation on a test set. Inference latency is the end-to-end processing time from input to output. For example, its unit includes milliseconds. Normalization is a preprocessing method to eliminate dimensional differences. For example, the original data can be linearly transformed to a uniform interval using the min-max normalization formula, such as mapping the temperature coefficient range of 0.1 to 10 to the interval [0,1].

[0085] The moving average filtering method is used to remove noise from the normalized data, and outliers are identified by box plot method.

[0086] Moving average filtering is a smoothing technique that calculates a weighted average using a sliding window. For example, a weighted average with a window size of 5 can be used, where the current value = 0.2 × current value + 0.18 × previous value + ..., to suppress high-frequency noise in the data. Box plot methods are statistical techniques based on the interquartile range (IQR) to identify outliers. For example, data exceeding (Q3 + 1.5IQR) or falling below (Q1 + 1.5IQR) are marked as outliers. For instance, in accuracy data with an IQR of 0.1 and a Q3 of 0.8, values ​​above 0.95 will be identified as outliers.

[0087] Based on the filtered data, a nonlinear mapping relationship between knowledge distillation parameters and student model performance parameters is established using the support vector machine algorithm.

[0088] Support Vector Machine (SVM) algorithms are classification and regression methods that use kernel tricks to handle nonlinear relationships. For example, a Radial Basis Function (RBF) kernel is used to map three-dimensional input parameters to a high-dimensional feature space. The three-dimensional input parameters include the temperature coefficient, knowledge weight coefficient, and feature alignment loss weight. The mapping relationship between parameter combinations and two-dimensional output metrics is established by solving for the maximum classification margin hyperplane, where the two-dimensional output metrics are accuracy and latency. In a specific embodiment, when the input parameters are temperature coefficient = 2, knowledge weight = 0.7, and feature alignment loss weight = 0.4, the model can output a prediction accuracy of 0.92 and a latency of 150ms. This process uses cross-validation to select optimal kernel parameters, such as the γ value of the RBF and the penalty coefficient C, to capture the combined effect of parameter interactions on performance, such as the coupling effect between the temperature coefficient and the feature alignment loss weight.

[0089] This embodiment collects real-time data of knowledge distillation parameters and performance indicators and performs normalization preprocessing to eliminate dimensional differences. Moving average filtering and box plot techniques are used to remove noise and outliers to improve data quality. Finally, a support vector machine algorithm is employed to establish a nonlinear mapping relationship between parameters and performance through a kernel method to capture complex interactive effects. This achieves the following technical effects: First, the combination of normalization and filtering eliminates dimensional differences and noise, while box plots filter extreme outliers, making the training data for the mapping model purer and reducing the risk of overfitting. Second, parameter quantification clarifies the independent and interactive effects on accuracy and latency; for example, it reveals that high feature alignment weights may improve accuracy but increase latency. Finally, the nonlinear decision boundary of the RBF kernel effectively handles the complex relationship between parameters and performance, which is more realistic than traditional linear methods and supports multi-objective optimization to guide parameter search, thus providing a reliable foundation for subsequent optimization and adaptive adjustment.

[0090] In one embodiment, the knowledge distillation parameters include the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight; the student model performance parameters include model accuracy and inference latency; and a nonlinear mapping relationship is constructed between the knowledge distillation parameters and the student model performance parameters. A Bayesian optimization algorithm is used for intelligent optimization, wherein the optimization steps include:

[0091] The three parameters of the teacher model—temperature coefficient, knowledge weight coefficient, and feature alignment loss weight—are encoded into real number vectors according to the preset parameter encoding method, and the vector dimension is set according to the parameter precision requirements.

[0092] The teacher model temperature coefficient is a parameter that controls the smoothness of the output probability distribution of the teacher model. Adjusting its value can change the confidence distribution during knowledge transfer. The knowledge weight coefficient is a parameter that adjusts the weighting ratio between the knowledge distillation loss and the student model's original classification loss. For example, setting it to 0.5 ensures equal weights for both classes. The feature alignment loss weight is a parameter that constrains the similarity of features in the intermediate layers of the student and teacher models. Feature differences can be calculated using mean squared error or cosine similarity. Parameter encoding and vectorization are processes that map discrete or continuous parameters into real-valued vectors. For example, when the temperature coefficient is set to a range of [1, 10], it can be encoded as a 10-dimensional vector representing different discrete values. The knowledge weight and feature alignment weight, as continuous parameters, are directly converted to floating-point numbers. The vector dimension is set according to the parameter precision requirements. For example, when the temperature coefficient needs to distinguish integer values ​​from 1 to 10, the dimension is set to 10 to achieve individual differentiation. This process unifies the multi-dimensional parameter space into a numerical form through parameter encoding rules, facilitating subsequent optimization algorithms.

[0093] Obtain the set real number vector, set the acquisition function of the Bayesian optimization algorithm, and use the normalization processing method to convert the model accuracy and inference latency into dimensionless indicators;

[0094] Model accuracy is the proportion of correctly predicted samples in a classification task, used to measure the model's inference precision. Inference latency is the response time from input to output, reflecting the model's real-time performance on edge devices. Normalization is a technique to eliminate differences in the dimensions of metrics; for example, accuracy remains unchanged, while inference latency is converted into a metric with the same direction as accuracy. This operation balances the weights of various metrics in multi-objective optimization through linear or nonlinear transformations, preventing the absolute value of inference latency from dominating the optimization direction and thus ensuring the rationality of the multi-objective optimization path.

[0095] After the acquisition function is set, a surrogate function is constructed using a Gaussian process regression model, and the Bayesian optimization algorithm is explored by setting the balance factor to a preset value, and the parameters are iteratively updated using a sequence optimization method.

[0096] Gaussian process regression models are probabilistic models used to construct surrogate functions, predicting the globally optimal region through historical sampling points. The exploration leverages a balancing factor, a parameter that controls the algorithm's choice between known high-yield regions and unknown potential regions; for example, setting a higher exploration weight can reduce the risk of getting trapped in local optima. Sequential optimization selects the next optimal sampling point by maximizing the acquisition function, such as a probability boosting function, and progressively updates the parameter combination. This process combines a surrogate model with a dynamic balancing mechanism to balance global search and local fine-tuning under limited computational resources, improving optimization efficiency and reducing the number of iterations.

[0097] Based on the iterative process of the Bayesian optimization algorithm, the initial number of sampling points is set to a preset value, and the maximum number of iterations is set to a preset value.

[0098] The initial number of sampling points is the number of parameter combinations randomly selected during the algorithm's startup phase; for example, setting it to 5 points allows for rapid coverage of the parameter space. The maximum number of iterations is one of the algorithm's termination conditions; for example, setting it to 50 iterations limits computational resource consumption. This setting, by pre-setting the initial distribution and termination threshold, avoids overfitting while preventing infinite iterations, ensuring the algorithm's practicality in engineering scenarios.

[0099] Once the Bayesian optimization algorithm reaches the convergence condition or the maximum number of iterations, the position vector corresponding to the optimal parameters is decoded and used as the target knowledge distillation parameters for the target performance index.

[0100] Convergence criteria are the standards for determining when an algorithm terminates, such as a performance improvement of less than a threshold after N consecutive iterations. The decoding process restores the optimal parameter vector to its original parameter form, such as discrete values ​​of the temperature coefficient. This operation ensures that the final parameters meet the actual constraints through reverse mapping, while preserving the optimization algorithm's compatibility with discrete parameters.

[0101] This embodiment transforms discrete parameters into high-dimensional vectors through parameter encoding rules to ensure compatibility with continuous optimization algorithms. Normalization ensures balanced optimization of multi-objective indices under a unified scale. A Gaussian process surrogate model combined with a sequence optimization strategy reduces the number of training iterations of the real model. A combination strategy of preset initial sampling and termination conditions balances computational efficiency and convergence reliability. Finally, the optimal parameters that conform to the actual constraints are obtained through the decoding process. This scheme resolves the contradiction between the discreteness and continuity of parameter space in traditional optimization, reduces the complexity of multi-objective optimization, and achieves efficient parameter optimization in computationally limited scenarios through the surrogate model and dynamic balancing mechanism, significantly improving the automation and adaptability of knowledge distillation parameter optimization.

[0102] In one embodiment, the optimized target knowledge distillation parameters are sent to the model training engine, student model performance parameters are collected in real time and the process is monitored, and re-optimization is triggered when the student model performance parameters deviate, including:

[0103] Obtain the optimized target knowledge distillation parameters and send them to the automatic adjustment module of the model training engine through a preset communication interface;

[0104] The pre-defined communication interface can be a standardized data transmission protocol between the training engine and the parameter optimization module, such as a gRPC-based remote procedure call interface or a message queue (like Kafka) to achieve real-time transmission of parameter configurations. This interface can achieve data transmission by serializing parameters into a specified format (such as JSON or Protobuf). The automatic tuning module can be a subsystem within the training engine responsible for receiving parameter configurations and dynamically adjusting the training process, typically including a parameter parser and a model configuration updater. This module can automate parameter configuration by parsing the received parameters and updating the model training configuration parameters, such as modifying the hyperparameter configuration file in the training script.

[0105] The teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight are controlled according to the target knowledge distillation parameters, and the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight are monitored in real time.

[0106] The teacher model temperature coefficient can be considered a temperature parameter used during the distillation process to soften the output probability distribution of the teacher model; a larger value indicates a smoother distribution. This parameter can be obtained by adjusting the temperature scaling factor in the loss function. The knowledge weight coefficient can be a weight parameter balancing the distillation loss and the original classification loss, for example, setting the knowledge distillation loss to account for 70% of the total loss. This coefficient can be achieved by setting a weight parameter in the loss function. The feature alignment loss weight can be a weight in the loss function that controls the similarity of the intermediate layer feature maps between the teacher model and the student model, typically using mean squared error or cosine similarity as alignment metrics. This weight can be obtained by setting a coefficient in the feature alignment loss function. Real-time monitoring can continuously record the actual values ​​of the parameters through sensors or log collection modules, for example, recording the current temperature coefficient and knowledge weight coefficient once per training batch. This process can be achieved by using a monitoring hook function to read and store the actual parameter values ​​in real time.

[0107] If the monitored actual parameters deviate from the target knowledge distillation parameters, an adaptive sliding mode control algorithm is used to dynamically adjust the input signal of the training hyperparameters.

[0108] Among these, the adaptive sliding mode control algorithm can be a robust control method that eliminates system uncertainties by switching between the control law and the adaptive law, for example, by designing the control input using Lyapunov stability theory. This algorithm can calculate the switching surface function and design the control input based on the Lyapunov function. ,in and For adaptive gain parameters, This represents an uncertainty estimate. The input signal for training the hyperparameters can be a control variable used to adjust the parameters, such as by modifying the current parameter values ​​through gradient correction terms or direct parameter overwriting. This process can be iteratively adjusted by adjusting the input signal, forcing the actual parameters to converge to the target value.

[0109] The performance monitor continuously collects data on model accuracy and inference latency, and the particle filter algorithm is used to obtain the optimal estimated student model performance parameters.

[0110] The performance monitor can be a lightweight monitoring component deployed in the training or inference environment, such as collecting accuracy and inference latency data for each batch via TensorBoard or a custom logging system. This component can achieve continuous data recording by setting a fixed collection frequency. The particle filtering algorithm can be a Monte Carlo-based filtering technique that estimates the true state by generating a swarm of parametric particles and weighting them, suitable for state estimation of nonlinear, non-Gaussian systems. This algorithm can initialize the parametric particle swarm, predict the particle state based on a motion model (such as a prediction model where parameters change with training epochs), calculate weights using an observation model, and finally obtain the optimal estimate through a weighted average (such as a Gaussian noise model).

[0111] The trend of the optimal estimated student model performance parameters is calculated using the exponentially weighted moving average method, and anomaly detection is performed by combining it with control chart methods.

[0112] Among them, the exponentially weighted moving average method can calculate the trend value of the time series by decaying weights, and the formula is: ,in This represents the current trend value. This represents the trend value from the previous moment. This is the smoothing coefficient. Control chart methods can be used as anomaly detection techniques in statistical process control, for example, setting the center line as the target performance value and the upper / lower control limits as target ± 3σ. This method compares the current trend value with the control limits, and determines an anomaly when three consecutive samples exceed the upper or lower limit.

[0113] If the model accuracy or inference latency exceeds the preset threshold of the target value for three consecutive samplings, the knowledge distillation parameter re-optimization procedure is triggered, and the optimal combination of knowledge distillation parameters is recalculated.

[0114] The preset threshold can be a performance boundary value set based on historical data statistics or domain knowledge. The triggering mechanism can be implemented by comparing three consecutive sampled values ​​with the control limit, such as when the accuracy is lower than 95% of the target value for three consecutive times. The re-optimization program can re-call the nonlinear mapping model established in step S100 and, combined with the current domain data characteristics, re-search the parameter space through an optimization algorithm to generate a new optimal parameter combination, such as an improved genetic algorithm.

[0115] This embodiment automates parameter configuration through a preset communication interface and an automatic adjustment module. The sliding mode control algorithm eliminates deviations caused by parameter drift, and particle filtering combined with the exponentially weighted moving average method provides accurate performance parameter estimation. The control chart and threshold triggering mechanism quickly respond to performance anomalies, and closed-loop feedback re-optimizes the parameter combination. This achieves the technical effect of significantly improving the reliability of parameter configuration, the accuracy of performance estimation, and the speed of dynamic response. At the same time, by integrating control theory and machine learning technology, a more robust closed-loop adjustment system is formed, thereby enhancing the environmental adaptability and engineering reliability of the entire lightweight method.

[0116] In one embodiment, model characteristics include model generalization ability and inference stability. During model inference, student model performance parameters are monitored, and model characteristics are managed to control scenarios where student model performance parameters fluctuate irregularly or become uncontrollable. This includes:

[0117] Statistical process control charts were used to monitor the changing trends of student model performance parameters. The statistical process control charts applied the eight outlier criteria of sigma control charts.

[0118] A statistical process control chart (SPC) is a tool used to determine whether a system process is under statistical control by plotting the distribution of data points against control limits. It can be used to monitor performance parameters in a student model inference process in real time, such as output confidence, inference latency, and energy consumption. This control chart sets control limits of ±3σ by calculating the mean and standard deviation of the parameters, and then overlays eight outlier detection criteria from a sigma control chart. These eight outlier detection criteria include, but are not limited to: a single data point exceeding the ±3σ control limit, nine consecutive points falling on the same side of the center line, six consecutive points increasing or decreasing, and 14 consecutive points fluctuating alternately. These criteria comprehensively judge whether performance parameters exhibit abnormal fluctuations through multi-dimensional statistical characteristics, such as point distribution trends and the number of outliers.

[0119] Determine whether the conditions for discrimination are met based on eight criteria for discrimination.

[0120] The determination of outlier conditions is achieved through multi-criteria logical judgment, such as simultaneously detecting whether data points exceed control limits, whether there is a continuous increasing or decreasing trend, or whether they exhibit periodic alternating fluctuations. This process combines outlier detection and trend analysis methods from statistical process control theory, and can identify complex anomaly patterns that may be missed by the traditional 3σ criterion, such as small-amplitude continuous shifts or periodic oscillations.

[0121] If the discrepancy criteria are not met, it is determined that the student model performance parameters do not exhibit irregular fluctuations or are uncontrollable, and the subsequent inference task continues.

[0122] When none of the discrepancy criteria are triggered, the system determines that the current inference process is in a stable state and the fluctuation of performance parameters is within the normal range. At this time, the inference process does not need to be interrupted or adjusted, and the execution of subsequent inference tasks can continue directly.

[0123] If the discrepancy criteria are met, it is determined that the student model performance parameters have irregular fluctuations or are uncontrollable, triggering the detection and analysis program for model characteristics, and feeding the detection and analysis results back to the model inference engine for adaptive compensation.

[0124] The detection and analysis program uses multi-dimensional data tracing to pinpoint the root cause of anomalies. This includes analyzing whether there are sudden changes in the input data distribution, abnormal hardware resources, or logical errors in the model inference path. Hardware resources include GPU utilization and memory bandwidth. Adaptive compensation dynamically adjusts the inference strategy based on the detection results. This could involve reducing model quantization precision to improve stability, enabling redundant computational paths to distribute the load, or dynamically adjusting confidence thresholds to avoid misjudgments. These compensation strategy adjustments are fed back to the inference process in real-time through the model inference engine, suppressing performance fluctuations and optimizing resource utilization. If the anomaly persists, the system will further trigger a higher-level model re-optimization mechanism, such as re-invoking the parameter re-optimization process.

[0125] This embodiment employs a statistical process control chart combined with eight anomaly detection criteria to achieve multi-dimensional anomaly detection. Multi-criteria logical judgment enhances the sensitivity and specificity of anomaly identification. Through a linkage mechanism between the detection and analysis program and adaptive compensation, the root cause of the problem can be quickly located and dynamic intervention implemented without interrupting the inference task. Simultaneously, hardware resource monitoring and inference path optimization strategies are combined to balance model stability and resource utilization. This scheme significantly reduces the risk of false positives or false negatives associated with traditional single anomaly detection criteria, shortens anomaly recovery time, enhances the robustness of the inference process, and avoids performance degradation due to overcompensation through dynamic adjustment strategies, ultimately achieving a synergistic improvement in model inference efficiency and stability.

[0126] In one embodiment, the target domain data characteristic data includes data distribution offset, domain semantic complexity, and user task suitability indicators. The target domain data characteristic data is acquired, and big data analysis is performed on the student model performance parameters, knowledge distillation parameters, and target domain data characteristic data during the training process. Furthermore, the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is continuously corrected and improved based on the inherent connections between the parameters, including:

[0127] Obtain distribution offset data, semantic complexity data, and user task suitability indicators for the target domain data, and obtain knowledge distillation parameters and student model performance parameters from the training system;

[0128] Outlier detection is performed on distribution offset data, semantic complexity data, user task fit index, knowledge distillation parameters and student model performance parameters. Outliers are identified using box plot method, and data exceeding the preset multiple interquartile range are smoothed.

[0129] Based on the smoothed data, principal component analysis was used to reduce dimensionality and extract the main feature variables.

[0130] Using the main feature variables, the FP-growth algorithm is used to perform association rule analysis to mine the association rules between knowledge distillation parameters, student model performance parameters and target domain data characteristics. The rule is that if the temperature coefficient is within a specified range and the knowledge weight coefficient is within a specified range, then the user task suitability score is above a specified value.

[0131] The adaptive response surface methodology is used to correct the model. New sampling points are generated through central composite experimental design. The weighted least squares method is used to fit the nonlinear mapping relationship between the new knowledge distillation parameters and the student model performance parameters.

[0132] Among these, data distribution offset can be a quantitative indicator measuring the difference in distribution between the target domain data and the teacher model training data. It can be obtained by calculating the distance in the feature space using the maximum mean difference or KL divergence. For example, data distribution offset can include the pixel distribution difference in the image domain or the word vector space distance in the text domain. Domain semantic complexity can be an indicator reflecting the inherent structural complexity of the target domain data. It can be calculated using features such as lexical diversity in the text domain or texture complexity in the image domain. For example, domain semantic complexity can include the TFIDF distribution entropy in the text domain or the texture gradient change rate in the image domain. User task fit index can be used to evaluate the actual performance of the student model in a specific task in the target domain, such as the mean accuracy in object detection or the F1 score in text classification. For example, user task fit index can include a weighted comprehensive score of inference latency and accuracy. Knowledge distillation parameters can be configuration parameters that control the knowledge transfer process, including temperature coefficients or loss weights. For example, knowledge distillation parameters can include the weighted fusion coefficients of the teacher and student model outputs in the distillation loss. Student model performance parameters can be quantitative indicators that characterize the efficiency and accuracy of lightweight model operation, including inference latency or accuracy. For example, student model performance parameters can include memory usage or computational throughput.

[0133] Box plotting can be a method for identifying outliers by calculating the interquartile range (IQR) of data. For example, the IQR can be calculated as the difference between the lower and upper quartiles of the dataset, and data exceeding 1.5 times the IQR can be marked as outliers. Moving averages or interpolation can then be used for smoothing. Principal component analysis (PCA) can be a method for projecting high-dimensional data into a low-dimensional space through linear transformation. For example, orthogonal principal components can be extracted through eigenvalue decomposition of the covariance matrix, retaining the feature variables that have the greatest impact on the target domain. FP-growth can be an unsupervised association rule analysis method for mining frequent itemsets. For example, a conditional frequent pattern base can be constructed and a frequent pattern tree can be recursively generated to discover the conditional dependencies between parameter combinations and user task fit. Adaptive response surface methodology can be a dynamic optimization method based on experimental design. For example, multidimensional parameter sampling points can be generated through central composite experimental design, and a nonlinear response surface model can be fitted using weighted least squares, where the weight allocation is based on the degree of influence of the parameters on user task fit.

[0134] This embodiment improves the reliability of input data by quantifying the characteristics of target domain data and combining it with outlier detection. It uses principal component analysis to extract key feature variables to reduce dimensionality complexity, and employs the FP-growth algorithm to mine the association rules between parameter combinations and task suitability as the basis for configuration. Finally, it constructs a dynamic response surface model through central composite experimental design and weighted least squares method, incorporating the nonlinear mapping relationship between domain characteristic parameters and model performance parameters into the adaptive optimization process. This can enhance the pertinence and robustness of knowledge distillation parameter configuration in cross-domain scenarios, while improving the actual deployment efficiency and performance assurance level of the lightweight model in specific tasks.

[0135] In one embodiment, historical distillation data under different target domain data characteristics is acquired. This historical distillation data includes combinations of historical target knowledge distillation parameters. A knowledge base and case base for optimizing knowledge distillation parameters are established, forming a standardized lightweight knowledge distillation scheme. Based on this standardized lightweight knowledge distillation scheme, adaptive matching and adjustment of the knowledge distillation parameters of the large domain model are performed, including:

[0136] Historical distillation data under different target domain data characteristics were obtained. The historical distillation data included combinations of historical target knowledge distillation parameters. K-means algorithm was used for cluster analysis.

[0137] Historical distillation data can be a dataset containing complete records of past distillation tasks, obtained through cross-domain task execution logs or experimental records. For example, historical distillation data may include data distribution characteristics of a specific domain, optimized combinations of knowledge distillation parameters, and the final performance metrics of the student model. The K-means algorithm is an unsupervised learning method that iteratively optimizes and divides data into K clusters. Its purpose is to identify potential grouping patterns of different domain parameter combinations and data characteristics in historical data, such as classifying similar domain features or parameter configurations into the same cluster. In a specific embodiment, by extracting parameter combinations and data characteristic vectors from historical distillation data, calculating the distance between data points and centroids, and dynamically adjusting the number of clusters until cluster stability converges, similar case groups are divided, providing an efficient indexing basis for subsequent case retrieval.

[0138] Outlier detection is performed on the data based on the cluster analysis results. Outliers are identified by box plot method, and outliers exceeding the preset multiple interquartile range are replaced with the mean.

[0139] Box plots are a method for identifying outliers based on the interquartile range (IQR) of data distribution. For example, box plots can mark values ​​exceeding 1.5 times the IQR as outliers. The IQR can be the difference between the third quartile (Q3) and the first quartile (Q1), reflecting the dispersion of the middle 50% of the data. In a specific embodiment, a box plot is drawn on the clustered dataset, calculating Q1, Q3, and IQR for each dimension. Outliers exceeding Q3+1.5IQR or below Q1+1.5IQR are marked, and mean replacement or interpolation methods are used to correct outliers, thereby eliminating noisy data and improving the robustness of subsequent modeling.

[0140] Establish a rule-based expert knowledge base, which includes rules in the form of if-then;

[0141] The if-then rule can be domain experience expressed in the form of a conditional statement. For example, an if-then rule could include "If the domain data has high noise, then increase the distillation temperature coefficient to 2.0". The expert knowledge base can be a structured database storing domain expert experience, containing parameter adjustment strategies, domain features, and the logic relating performance metrics. In a specific embodiment, by transforming the practical experience of domain experts into a rule set, such as "If the target domain data has class imbalance, then set the classification loss weight to the inverse of the class ratio", the rule base quickly filters applicable rules when a new task is input through a conditional matching mechanism, providing prior constraints for parameter recommendation.

[0142] A case library is constructed using a case-based reasoning algorithm. The case library includes data characteristics of the target domain, combinations of knowledge distillation parameters, and actual reasoning results. The similarity between cases is calculated using cosine similarity.

[0143] Cosine similarity is an index that measures the cosine of the angle between two vectors in a high-dimensional space, with a value range of [1,1]. It is used to evaluate the similarity of data characteristics and parameter configurations between cases. The case library structure can be a multi-dimensional feature set storing historical cases. For example, the case library can include dimensions such as target domain data characteristics, knowledge distillation parameter combinations, and actual inference performance. Target domain data characteristics include the number of categories and feature dimensions; knowledge distillation parameter combinations include temperature coefficient and loss weight; and actual inference performance includes accuracy and latency. In a specific embodiment, each historical case is represented as a vector containing the above features. When new domain data is input, its feature vector is extracted and cosine similarity is calculated with all cases in the case library. The Top N similar cases are selected by similarity ranking, thereby quantifying similarity and quickly locating the optimal reference case.

[0144] The system receives data characteristics input from the target domain, performs rule-based reasoning based on the expert knowledge base, retrieves similar cases from the case library, and obtains target knowledge distillation parameters by combining fuzzy comprehensive evaluation methods. Based on the target knowledge distillation parameters, it performs adaptive matching and adjustment on the domain-wide model.

[0145] The fuzzy comprehensive evaluation can be a multi-criteria decision-making method based on fuzzy set theory, which uses a membership function to weight and fuse the matching results between the rule base suggestions and the case base. The case reasoning algorithm can be a method that derives parameter configurations using parameter combinations of similar cases and conditional constraints of the rule base. In a specific embodiment, after inputting the characteristics of new domain data, preliminary parameter suggestions are first generated through rule matching in the expert knowledge base, such as a temperature coefficient ≥ 1.5. At the same time, the case base retrieves parameter combinations of similar cases and calculates weights, such as similarity scores. The rule constraints and case parameters are then fused according to preset weights through fuzzy comprehensive evaluation, ultimately outputting target parameters that balance domain characteristics and historical experience. For example, if the rule suggestion temperature coefficient is 1.8, and the average value of similar cases is 1.6, the comprehensive evaluation might take 1.7 as the final parameter.

[0146] This embodiment achieves potential grouping pattern recognition through cluster analysis of historical distillation data, improves data quality by combining box plot method and interquartile range to eliminate noise data, forms dual constraints by using cosine similarity matching of if-then rule base and case base, and finally generates target parameters by fusing multi-source knowledge through fuzzy comprehensive evaluation. This significantly improves the automation level and cross-domain generalization ability of knowledge distillation parameter configuration, and solves the problems of traditional methods relying on human experience and low efficiency in cross-domain transfer.

[0147] Furthermore, this embodiment of the invention also proposes a storage medium storing a lightweight adaptive program for a domain large model based on knowledge distillation. When the lightweight adaptive program for a domain large model based on knowledge distillation is executed by a processor, it implements the steps of the lightweight adaptive method for a domain large model based on knowledge distillation as described above.

[0148] In addition, refer to Figure 3 This invention also proposes a lightweight adaptive system for large domain models based on knowledge distillation, which includes:

[0149] The parameter modeling module 10 is used to obtain the knowledge distillation parameters and student model performance parameters of the domain large model, and to establish a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters.

[0150] The parameter optimization module 20 is used to optimize the best combination of knowledge distillation parameters based on the nonlinear mapping relationship between the constructed knowledge distillation parameters and the student model performance parameters, so as to obtain the target knowledge distillation parameters that achieve the target performance index.

[0151] The monitoring and adjustment module 30 is used to send the optimized target knowledge distillation parameters to the model training engine, collect student model performance parameters in real time and monitor the process, and trigger re-optimization when the student model performance parameters deviate.

[0152] The feature control module 40 is used to monitor the performance parameters of the student model during the model inference process and to control the model features in scenarios where the performance parameters of the student model fluctuate irregularly or are uncontrollable.

[0153] The model correction module 50 is used to acquire target domain data characteristic data, perform big data analysis on student model performance parameters, knowledge distillation parameters and target domain data characteristic data during the training process, and continuously correct and improve the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters based on the inherent relationship between the parameters.

[0154] The knowledge adaptation module 60 is used to acquire historical distillation data under different target domain data characteristics. The historical distillation data includes historical target knowledge distillation parameter combinations, establishes a knowledge base and case base for optimizing knowledge distillation parameters, forms a standardized lightweight knowledge distillation scheme, and adaptively matches and adjusts the knowledge distillation parameters of the domain large model based on the standardized lightweight knowledge distillation scheme.

[0155] Other embodiments or specific implementations of the lightweight adaptive system for large domain models based on knowledge distillation described in this invention can be found in the above-described method embodiments, and will not be repeated here.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments. In the module claims listing several systems, several of these systems may be specifically embodied by the same hardware item. The use of the terms first, second, and third, etc., does not indicate any order and can be interpreted as names.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as a read-only memory image (ROM) / random access memory (RAM), magnetic disk, optical disk), and includes several instructions to cause a terminal user device (which may be a mobile phone, computer, server, air conditioner, or network user device, etc.) to execute the methods described in the various embodiments of the present invention.

[0159] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A lightweight adaptive method for large domain models based on knowledge distillation, characterized in that, The method is applied to both image and text domains, and includes: Obtain the knowledge distillation parameters and student model performance parameters of the domain-wide model, and establish a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters; Based on the nonlinear mapping relationship between the constructed knowledge distillation parameters and the student model performance parameters, the optimal combination of knowledge distillation parameters is optimized to obtain the target knowledge distillation parameters that achieve the target performance index. The optimized target knowledge distillation parameters are sent to the model training engine, and the student model performance parameters are collected in real time and the process is monitored. When the student model performance parameters deviate, re-optimization is triggered. During the model inference process, the performance parameters of the student model are monitored, and the model characteristics are controlled in scenarios where the performance parameters of the student model fluctuate irregularly or are uncontrollable. Acquire target domain data characteristics, perform big data analysis on student model performance parameters, knowledge distillation parameters, and target domain data characteristics during the training process, and continuously correct and improve the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters based on the inherent relationship between each parameter; Historical distillation data under different target domain data characteristics is obtained. The historical distillation data includes combinations of historical target knowledge distillation parameters. A knowledge base and case base for optimizing knowledge distillation parameters are established to form a standardized lightweight knowledge distillation scheme. Based on the standardized lightweight knowledge distillation scheme, the knowledge distillation parameters of the domain large model are adaptively matched and adjusted. The target domain data characteristic data includes data distribution offset, domain semantic complexity, and user task suitability indicators. The process of acquiring the target domain data characteristic data, performing big data analysis on the student model performance parameters, knowledge distillation parameters, and target domain data characteristic data during the training process, and continuously correcting and improving the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters based on the inherent connections between the parameters includes: The system acquires distribution offset data, semantic complexity data, and user task fit index of the target domain data, and obtains knowledge distillation parameters and student model performance parameters from the training system. The data distribution offset includes pixel distribution difference in the image domain or word vector space distance in the text domain. The domain semantic complexity includes TFIDF distribution entropy in the text domain or texture gradient change rate in the image domain. The user task fit index includes mean accuracy in the object detection task or F1 score in the text classification task. Outlier detection is performed on the distribution offset data, the semantic complexity data, the user task suitability index, the knowledge distillation parameters, and the student model performance parameters. Outliers are identified using box plots, and data exceeding a preset multiple of the interquartile range are smoothed. Based on the smoothed data, principal component analysis was used to reduce dimensionality and extract the main feature variables. Using the main feature variables, the FP-growth algorithm is used to perform association rule analysis to mine the association rules between the knowledge distillation parameters, the student model performance parameters and the data characteristics of the target domain, and to obtain the rule that if the temperature coefficient is within a specified range and the knowledge weight coefficient is within a specified range, then the user task suitability score is above a specified value. The model is modified using the adaptive response surface methodology. New sampling points are generated through a central composite experimental design. The nonlinear mapping relationship between the new knowledge distillation parameters and the student model performance parameters is fitted using the weighted least squares method.

2. The lightweight adaptive method for large domain models based on knowledge distillation as described in claim 1, characterized in that, The knowledge distillation parameters include the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight; the student model performance parameters include model accuracy and inference latency; and the process of obtaining the knowledge distillation parameters of the domain-wide model and the student model performance parameters, and establishing a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters, includes: Real-time data of the teacher model temperature coefficient, the knowledge weight coefficient, and the feature alignment loss weight are collected, and the model accuracy and inference latency are measured. The collected raw data are then normalized. The moving average filtering method is used to remove noise from the normalized data, and outliers are identified by box plot method. Based on the filtered data, a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters is established using the support vector machine algorithm.

3. The lightweight adaptive method for large domain models based on knowledge distillation as described in claim 1, characterized in that, The knowledge distillation parameters include the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight. The student model performance parameters include model accuracy and inference latency. The nonlinear mapping relationship between the constructed knowledge distillation parameters and the student model performance parameters is intelligently optimized using a Bayesian optimization algorithm, wherein the optimization steps include: The three parameters—the teacher model temperature coefficient, the knowledge weight coefficient, and the feature alignment loss weight—are encoded into a real number vector according to a preset parameter encoding method, and the vector dimension is set according to the parameter precision requirements. Obtain the set real number vector, set the acquisition function of the Bayesian optimization algorithm, and use the normalization processing method to convert the model accuracy and the inference delay into dimensionless indicators; After the acquisition function is set, a surrogate function is constructed using a Gaussian process regression model, and the Bayesian optimization algorithm exploration - using the balance factor as a preset value, is used to iteratively update the parameters using a sequence optimization method; Based on the iterative process of the Bayesian optimization algorithm, the initial number of sampling points is set to a preset value, and the maximum number of iterations is set to a preset value. Once the Bayesian optimization algorithm reaches the convergence condition or the maximum number of iterations, the position vector corresponding to the optimal parameters is decoded and used as the target knowledge distillation parameters for the target performance index.

4. The lightweight adaptive method for large domain models based on knowledge distillation as described in claim 1, characterized in that, The optimized target knowledge distillation parameters are sent to the model training engine, student model performance parameters are collected in real time and the process is monitored, and re-optimization is triggered when the student model performance parameters deviate, including: Obtain the optimized target knowledge distillation parameters and send the target knowledge distillation parameters to the automatic adjustment module of the model training engine through a preset communication interface; The teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight are controlled according to the target knowledge distillation parameters, and the teacher model temperature coefficient, knowledge weight coefficient, and feature alignment loss weight are monitored in real time. If the monitored actual parameters deviate from the target knowledge distillation parameters, an adaptive sliding mode control algorithm is used to dynamically adjust the input signal of the training hyperparameters. The performance monitor continuously collects data on model accuracy and inference latency, and the particle filter algorithm is used to obtain the optimal estimated student model performance parameters. The trend of the optimal estimated student model performance parameters is calculated using the exponentially weighted moving average method, and anomaly detection is performed by combining it with control chart methods. If the model accuracy or inference latency exceeds the preset threshold of the target value for three consecutive samplings, the knowledge distillation parameter re-optimization procedure is triggered, and the optimal combination of knowledge distillation parameters is recalculated.

5. The lightweight adaptive method for large domain models based on knowledge distillation as described in claim 1, characterized in that, The model characteristics include model generalization ability and inference stability. The monitoring of student model performance parameters during model inference and the control of model characteristics for scenarios where student model performance parameters fluctuate irregularly or become uncontrollable include: Statistical process control charts are used to monitor the changing trends of student model performance parameters. The statistical process control charts apply the eight outlier criteria of sigma control charts. Determine whether the discrimination conditions are met based on the eight discrimination criteria described above; If the discrepancy criteria are not met, it is determined that the student model performance parameters do not exhibit irregular fluctuations or are uncontrollable, and the subsequent inference task continues. If the discrepancy criteria are met, it is determined that the student model's performance parameters exhibit irregular fluctuations or are uncontrollable, triggering a detection and analysis program for the model's characteristics. The detection and analysis results are then fed back to the model inference engine for adaptive compensation.

6. The lightweight adaptive method for large domain models based on knowledge distillation as described in claim 1, characterized in that, The process involves acquiring historical distillation data under different target domain data characteristics, including historical target knowledge distillation parameter combinations, establishing a knowledge base and case base for knowledge distillation parameter optimization, forming a standardized lightweight knowledge distillation scheme, and adaptively matching and adjusting the knowledge distillation parameters of the domain-wide model based on the standardized lightweight knowledge distillation scheme, including: Historical distillation data under different target domain data characteristics are obtained. The historical distillation data includes combinations of historical target knowledge distillation parameters. K-means algorithm is used for cluster analysis. Outlier detection is performed on the data based on the cluster analysis results. Outliers are identified by box plot method, and outliers exceeding the preset multiple interquartile range are replaced with the mean. Establish a rule-based expert knowledge base, which includes rules in the form of if-then; A case library is constructed using a case-based reasoning algorithm. The case library includes target domain data characteristics, knowledge distillation parameter combinations, and actual reasoning effects. The similarity between cases is calculated using cosine similarity. The system receives target domain data characteristics as input, performs rule-based reasoning based on the expert knowledge base, retrieves similar cases from the case library, and obtains target knowledge distillation parameters by combining fuzzy comprehensive evaluation method. Based on the target knowledge distillation parameters, it performs adaptive matching and adjustment on the domain large model.

7. A lightweight adaptive system for large domain models based on knowledge distillation, characterized in that, The system is applied to both image and text domains, and the system includes: The parameter modeling module is used to obtain the knowledge distillation parameters and student model performance parameters of the domain large model, and to establish a nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters; The parameter optimization module is used to optimize the best combination of knowledge distillation parameters based on the nonlinear mapping relationship between the constructed knowledge distillation parameters and the student model performance parameters, so as to obtain the target knowledge distillation parameters that achieve the target performance index. The monitoring and adjustment module is used to send the optimized target knowledge distillation parameters to the model training engine, collect student model performance parameters in real time and monitor the process, and trigger re-optimization when the student model performance parameters deviate. The feature control module is used to monitor the performance parameters of the student model during the model inference process and to control the model features in scenarios where the performance parameters of the student model fluctuate irregularly or are uncontrollable. The model correction module is used to acquire target domain data characteristic data, perform big data analysis on student model performance parameters, knowledge distillation parameters and target domain data characteristic data during the training process, and continuously correct and improve the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters based on the inherent relationship between the parameters. The knowledge adaptation module is used to acquire historical distillation data under different target domain data characteristics. The historical distillation data includes historical target knowledge distillation parameter combinations. It establishes a knowledge base and case base for optimizing knowledge distillation parameters, forms a standardized lightweight knowledge distillation scheme, and adaptively matches and adjusts the knowledge distillation parameters of the domain large model based on the standardized lightweight knowledge distillation scheme. The target domain data characteristic data includes data distribution offset, domain semantic complexity, and user task suitability indicators. The process of acquiring the target domain data characteristic data, performing big data analysis on the student model performance parameters, knowledge distillation parameters, and target domain data characteristic data during the training process, and continuously correcting and improving the nonlinear mapping relationship between the knowledge distillation parameters and the student model performance parameters based on the inherent connections between the parameters includes: The system acquires distribution offset data, semantic complexity data, and user task fit index of the target domain data, and obtains knowledge distillation parameters and student model performance parameters from the training system. The data distribution offset includes pixel distribution difference in the image domain or word vector space distance in the text domain. The domain semantic complexity includes TFIDF distribution entropy in the text domain or texture gradient change rate in the image domain. The user task fit index includes mean accuracy in the object detection task or F1 score in the text classification task. Outlier detection is performed on the distribution offset data, the semantic complexity data, the user task suitability index, the knowledge distillation parameters, and the student model performance parameters. Outliers are identified using box plots, and data exceeding a preset multiple of the interquartile range are smoothed. Based on the smoothed data, principal component analysis was used to reduce dimensionality and extract the main feature variables. Using the main feature variables, the FP-growth algorithm is used to perform association rule analysis to mine the association rules between the knowledge distillation parameters, the student model performance parameters and the data characteristics of the target domain, and to obtain the rule that if the temperature coefficient is within a specified range and the knowledge weight coefficient is within a specified range, then the user task suitability score is above a specified value. The model is modified using the adaptive response surface methodology. New sampling points are generated through a central composite experimental design. The nonlinear mapping relationship between the new knowledge distillation parameters and the student model performance parameters is fitted using the weighted least squares method.

8. A lightweight adaptive device for large domain models based on knowledge distillation, characterized in that, The knowledge distillation-based lightweight adaptive domain large model includes: a memory, a processor, and a knowledge distillation-based lightweight adaptive domain large model program stored in the memory and executable on the processor. The knowledge distillation-based lightweight adaptive domain large model program is configured to implement the steps of the knowledge distillation-based lightweight adaptive domain large model method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a lightweight adaptive program for a large domain model based on knowledge distillation. When the processor executes the lightweight adaptive program for a large domain model based on knowledge distillation, it implements the steps of the lightweight adaptive method for a large domain model based on knowledge distillation as described in any one of claims 1 to 6.

Images