Knowledge Distillation-based Lightweight Model Transfer Learning Method

Abstract

The present invention discloses a lightweight model transfer learning method based on knowledge distillation, belonging to the field of artificial intelligence security technology. The learning method includes: S1, constructing a security initialization module and deploying a teacher model on a preset cloud server; S2, generating a lightweight student model, where a preset processing module receives the teacher model and performs model compression processing on the teacher model; S3, knowledge transfer security protection, training the lightweight student model; S4, securely binding the deployment environment, binding and verifying the trained lightweight student model with a preset target operating device; S5, full-cycle audit tracking, establishing a traceable audit management module to record the full-process operation logs from the compression of the teacher model to the operation of the deployed lightweight student model. Through steps S1-S5, the present invention avoids the leakage of the original teacher model structure information and performs dynamic permission control on the teacher model in the deployment link.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence security, and particularly to a lightweight model transfer learning method based on knowledge distillation. Background Art

[0002] The lightweight model transfer learning method based on knowledge distillation is a method that transfers the knowledge of a large model to a small model to achieve model lightweight and high performance. The core idea of knowledge distillation is to use a powerful "teacher" model to guide a smaller "student" model to learn, so that the student model can have performance similar to that of the teacher model.

[0003] Since knowledge distillation can extract useful knowledge from the original deep learning model, but training the student model through knowledge distillation to make the performance of the student model reach or even exceed that of the teacher model, this model training method may also infringe the intellectual property rights of the teacher model. Existing protection methods for knowledge distillation lightweight models generally include: watermarking, model encryption, replacement and confusion of model weight information, etc.

[0004] In the existing lightweight model transfer learning method based on knowledge distillation, traditional knowledge distillation ignores security protection, and the obtained student model after compression is easy to obtain sensitive information through reverse engineering, resulting in the leakage of the original teacher model structure information, and there is a risk of unauthorized access in the deployment link of the teacher model.

[0005] Therefore, it is urgent to provide a lightweight model transfer learning method based on knowledge distillation to solve the above problems. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the above-mentioned disadvantages of the prior art and provide a lightweight model transfer learning method based on knowledge distillation.

[0007] To solve the above technical problem, a technical solution adopted by the present invention is: to provide a lightweight model transfer learning method based on knowledge distillation, including the following steps:

[0008] S1. Construction of a security initialization module: Deploy a teacher model on a preset cloud server, and establish a dynamic permission verification mechanism. After authorizing a user to pass the verification of the dynamic permission verification mechanism, the user can call the teacher model through a preset device and transmit it.

[0009] S2. Generation of a lightweight student model: A preset processing module receives the teacher model, performs model compression processing on the teacher model, adds protection marks to the key parameters generated during the compression process, and controllably interferes with the key knowledge fragments in the teacher model according to the protection marks to generate a lightweight student model.

[0010] S3. Knowledge transfer security protection: Train the lightweight student model, and verify data access permissions in real time during the training process, and dynamically adjust the protection intensity according to the training stage;

[0011] S4. Deployment environment security binding: Bind and verify the trained lightweight student model with a preset target operating device, and deploy a real-time behavior monitoring module to monitor abnormal operations of the lightweight student model during loading and inference;

[0012] S5. Full-cycle audit tracking: Establish a traceable audit management module, record the full-process operation logs from the compression of the teacher model to the operation of the deployed lightweight student model, generate non-tamperable verification marks for key operation nodes in the full-process operation logs, and generate a security assessment report regularly.

[0013] The present invention is further configured that: The dynamic permission verification mechanism in the step S1 includes the following contents:

[0014] S11. When the cloud server receives a device call request from an authorized user, detect whether the IP address, GPS location, and device ID of the authorized user's device are within a pre-authorized whitelist through a preset verification unit;

[0015] S12. If the verification is passed, the cloud server generates a one-time dynamic token, and matches the one-time dynamic token with the fingerprint information entered in the authorized user's device. After successful matching, transmit the one-time dynamic token to the authorized user's device;

[0016] S13. After receiving the one-time dynamic token, the authorized user's device performs local verification to ensure that the one-time dynamic token matches the fingerprint information entered in the authorized user's device correctly, and establishes a connection with the cloud server through a secure channel, and encrypts and transmits the teacher model to the processing module.

[0017] The present invention is further configured that: When the authorized user calls the teacher model in the step S1, adjust the permission level according to the behavior score of the authorized user. If the score is lower than a preset threshold, lower the permission level and limit the model access frequency; if the score is higher than the threshold, raise the permission level and increase the model access permission, and regenerate a one-time dynamic token after the permission change to ensure the uniqueness of the one-time dynamic token for each permission change and prevent unauthorized access; if the score is equal to the threshold, keep the permission level unchanged.

[0018] The present invention is further configured that: The operation permission levels of the authorized user are divided into: model query, parameter reading, and full call;

[0019] The behavior scores of the authorized users include: authentication behavior scores, operation behavior scores, device environment change scores, and data interaction behavior scores.

[0020] The present invention is further configured such that the steps for generating the lightweight student model in step S2 include:

[0021] S21. Perform hierarchical analysis on the network structure of the teacher model through the processing module, identify and remove duplicate or redundant components, and generate a simplified basic model framework;

[0022] S22. In the basic model framework, screen and merge parameter modules of the teacher model with similar functions, reduce the overall number of parameters of the teacher model, and retain the key parameters required for core knowledge transfer in the teacher model;

[0023] S23. Add recognizable protection marks to the key parameters, add controllable interference to the key knowledge fragments in the teacher model according to the protection marks, and finally generate a lightweight student model.

[0024] The present invention is further configured such that the steps for adding recognizable protection marks to the key parameters in step S23 are as follows:

[0025] S231. According to the functional importance of the parameters in the teacher model, screen out the key parameters that affect the core knowledge transfer in the teacher model, generate multiple parameter files, and assign a unique identifier to each key parameter in each parameter file;

[0026] S232. Embed the identifier into the description field of the parameter file so that the identifier corresponds one-to-one with the key parameter values in the parameter file to form a mark, generate a verification code for the marked parameter file, and store the verification code independently in a preset security verification module;

[0027] The marks of the key parameters in the parameter file are divided into: parameters with high importance marks, parameters with medium importance marks, and parameters with low importance marks;

[0028] The specific steps for adding controllable interference to the key knowledge fragments in the teacher model according to the protection marks in step S23 are as follows:

[0029] S233. For the parameters with high importance marks, superimpose random noise and limit the noise intensity not to exceed 15% of the original parameter value; for the parameters with medium importance marks, perform interval translation or proportional scaling on the parameter values, and control the offset amplitude within 5% - 10%; for the parameters with low importance, retain the original value and only add a small amount of perturbation.

[0030] S234. Generate an interference factor based on the hardware characteristics of the device of the authorized user, and the interference factor is automatically updated every preset period.

[0031] The present invention is further configured that: the specific content of knowledge transfer security protection in the step S3 includes:

[0032] S31. Set security constraint rules for data input and model weight update during training according to the key parameters and protection marks in the lightweight student model, and bind a preset permission verification interface;

[0033] S32. Divide the training process into multiple training cycles. Before the start of each training cycle, call the permission verification interface to verify the data access permission of the device of the authorized user. If the permission is invalid, suspend the training. If the verification passes, load the corresponding training data;

[0034] S33. Dynamically adjust the protection intensity for the weight update operation of the lightweight student model according to the training cycle and the sensitivity level of the training data, including adding an encryption verification layer for increasing the gradient update frequency and restricting the input scale of single training data.

[0035] The present invention is further configured that: the specific steps of security binding of the deployment environment in the step S4 include:

[0036] S41. Perform a binding verification between the trained lightweight student model and the target running device. If the target running device is the device of the authorized user, no verification is required, and the behavior monitoring module is used to monitor abnormal operations during the loading and inference processes of the lightweight student model. If the target running device is not the device of the authorized user, jump to step S42;

[0037] S42. The device of the authorized user sends a temporary verification key containing a time stamp to the target running device and requests to complete device identity matching through the authentication information of the authorized user within a preset time. If the target running device fails to complete the verification within the preset time, transfer the lightweight student model to an isolated storage area, and generate a security warning log containing the device identifier of the target running device, the reason for verification failure, and the operation time. If the verification is completed, the behavior monitoring module is used to monitor abnormal operations during the loading and inference processes of the lightweight student model.

[0038] The present invention is further configured that: the specific steps of monitoring abnormal operations during the loading and inference processes of the lightweight student model in the step S41 are as follows:

[0039] S411. When the lightweight student model is loaded onto the target operating device, the hash value of the model file of the lightweight student model is compared with the pre-stored standard hash value in real time. If they are inconsistent, it is determined that the model has been tampered with, the loading is interrupted, and a tampering warning is generated. If they are consistent, the comparison continues in real time;

[0040] S412. During the inference process of the lightweight student model, the format, numerical range, and source of the input data are verified in real time. If it is detected during the verification that the data exceeds the preset threshold or an unauthorized data input request, the data is restricted from entering the inference process and marked as abnormal input;

[0041] S413. When the behavior monitoring module detects tampering, abnormal input, or unauthorized model call behavior during the loading or inference process, the current operation is immediately terminated, the running state of the lightweight student model is frozen, and a security log containing the type of anomaly, trigger time, and operation context is generated and uploaded to the audit management module synchronously.

[0042] The present invention is further configured such that: the key operation nodes in the step S5 include teacher model compression, lightweight student model training, and deployment operations;

[0043] In the step S5, non-tamperable verification marks are generated for the key operation nodes in the full-process operation log, and the specific content includes:

[0044] S51. For the key operation nodes in the full-process operation log, the operation content, timestamp, and operator identity information are extracted, and the corresponding encrypted digest is generated through a preset one-way hash function, and the encrypted digest is bound and stored with the node data;

[0045] S52. The encrypted digests of each of the key operation nodes are concatenated in the operation time sequence. Each time a new key operation node is added, the encrypted digest of the previous key operation node is used as an input parameter to participate in the hash calculation of the current node, forming a chain-associated verification mark sequence.

[0046] The beneficial effects of the present invention are as follows:

[0047] 1. Through the dual authentication mechanism of device multi-dimensional information whitelist verification and one-time dynamic token binding to the device fingerprint, the present invention ensures that the teacher model is only called within the authorized device and geographical scope, preventing unauthorized access and man-in-the-middle attacks; combined with dynamic adjustment of permission levels and one-time token updates, it responds to operation risks in real time, effectively blocking attempts of unauthorized access and reverse engineering, and protecting the intellectual property rights of the model;

[0048] 2. By grading and marking the importance of parameters and introducing differential controllable interference, the present invention destroys the original feature correlation of key parameters while preserving the functional effectiveness of the student model, making it impossible for attackers to restore the teacher model structure through reverse engineering. Combined with the dynamic interference factor bound to the device characteristics, the difficulty of model stealing and reproduction is further increased, and lightweight deployment suitable for high-security requirement scenarios is achieved.

[0049] 3. Through chain encryption digest and irreversible hash calculation, the present invention generates an immutable verification tag sequence for key operation nodes to ensure that any data tampering at a node will cause global verification to fail. Through the standardized storage of operation logs, the separate management of algorithm parameters, and the real-time blocking of abnormal behaviors, full-process traceability is achieved, meeting the compliance audit requirements of scenarios such as medical and financial fields, while preventing data leakage and malicious tampering during the training and deployment processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 is the flowchart of the present invention;

[0051] Figure 2 is the flowchart of the dynamic permission verification mechanism of the present invention;

[0052] Figure 3 is the flowchart of the generation of the lightweight student model of the present invention;

[0053] Figure 4 is the flowchart of the knowledge transfer security protection of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] The following elaborates on the preferred embodiments of the present invention in conjunction with the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making the protection scope of the present invention more clearly defined.

[0055] Please refer to Figures 1-4 , the lightweight model transfer learning method based on knowledge distillation includes the following steps:

[0056] S1. Construction of the security initialization module. Deploy the teacher model on a preset cloud server (the structure of the teacher model includes the model architecture, such as the structure of a deep neural network, and also includes the parameter scale, such as the number of layers, the number of parameters, the dimension of the hidden layer, etc., which are all prior arts). Establish a dynamic permission verification mechanism. After being verified by the dynamic permission verification mechanism, an authorized user can call the teacher model through a preset device and transmit it. The cloud server is a distributed cluster architecture composed of multiple physical nodes, configured with a load balancing module and a redundant storage module to ensure the high availability and disaster recovery backup of the teacher model. An encrypted storage partition is deployed in the cloud server to store the teacher model file and associated metadata, and an access control list is set to restrict direct access by unauthorized devices;

[0057] Among them, the dynamic permission verification mechanism in step S1 includes the following:

[0058] S11. When the cloud server receives a device call request from an authorized user, it detects whether the IP address, GPS location, and device ID of the authorized user's device are within the pre-authorized whitelist through a preset verification unit;

[0059] S12. If the verification passes, the cloud server generates a one-time dynamic token, matches the one-time dynamic token with the fingerprint information entered in the authorized user's device, and transmits the one-time dynamic token to the authorized user's device after successful matching;

[0060] After receiving the one-time dynamic token, the authorized user's device performs local verification to ensure that the one-time dynamic token matches the fingerprint information entered in the authorized user's device without error, and establishes a connection with the cloud server through a secure channel, and encrypts and transmits the teacher model to the processing module. The specific content of the encrypted transmission is to encrypt the teacher model data in blocks using a symmetric encryption algorithm and encrypt and transmit the symmetric key using an asymmetric encryption algorithm to ensure double protection of data and key.

[0061] Among them, when the authorized user calls the teacher model in step S1, the permission level is adjusted according to the authorized user's behavior score. If the score is lower than the preset threshold, the permission level is lowered and the model access frequency is restricted; if the score is higher than the threshold, the permission level is raised and the model access permission is increased, and a new one-time dynamic token is generated after the permission change to ensure the uniqueness of the one-time dynamic token for each permission change and prevent unauthorized access; if the score is equal to the threshold, the permission level remains unchanged. The permission adjustment process is recorded in the full-process operation log.

[0062] Among them, the operation permission levels of the authorized user are divided into: model query, parameter reading, and full call;

[0063] The behavior scores of the authorized user include: authentication behavior score, operation behavior score, device environment change score, and data interaction behavior score.

[0064] Example 1 A teacher in a school calls the teacher model through an authorized mobile terminal to assist teaching. The initial permission is "full call". Because the teacher uses a new device when on a business trip or traveling (triggering a decrease in the device environment change score) and accidentally touches non-conventional parameters multiple times (decreasing the operation behavior score), the system lowers his permission to "parameter reading", only allowing viewing of the model output results but prohibiting modification. After the permission change, a new one-time dynamic token is generated, and the operation record is synchronously written into the audit log.

[0065] Fine-grained access control and real-time risk response are achieved by dynamically adjusting permission levels through behavioral scoring and binding unique one-time dynamic tokens. Combined with multi-dimensional device verification and audit tracking, unauthorized operations and data leakage are effectively prevented, ensuring the security and compliance of model calls in sensitive scenarios.

[0066] S2, lightweight student model generation, the preset processing module receives the teacher model, performs model compression processing on the teacher model, and adds protection marks to the key parameters generated in the compression process, and controllably interferes with the key knowledge fragments in the teacher model according to the protection marks to generate a lightweight student model;

[0067] The steps of generating the lightweight student model in step S2 include:

[0068] S21. Perform hierarchical analysis on the network structure of the teacher model through the processing module, identify and remove repeated or redundant components, generate a simplified basic model framework, and traverse each layer of the teacher model to obtain the contribution of the feature map output by each layer in the overall model prediction (the contribution refers to the degree of influence of the feature map of a certain layer on the final prediction result of the model. The gradient or activation value statistics of the layer output are calculated by back propagation to quantify its importance to the classification / regression task. For example, the larger the gradient or the more significant the activation, the higher the contribution), and perform structural similarity analysis on the layers that meet the contribution standards, and remove the layers marked as redundant or repeated (for the layers that meet the contribution standards, the structural similarity analysis determines the redundancy by comparing the parameter distribution or feature response between layers. For example, if the parameters of adjacent layers are highly similar, they are marked as redundant layers and only one of them is retained);

[0069] S22. In the basic model framework, filter and merge the parameter modules of the teacher model with similar functions, reduce the number of overall parameters of the teacher model, and retain the key parameters required for the core knowledge transfer in the teacher model;

[0070] S23. Add identifiable protection marks to key parameters, and add controllable interference to key knowledge fragments in the teacher model based on the protection marks, so that the teacher model cannot reversely restore the original features while retaining the functional effectiveness, and finally generate a lightweight student model.

[0071] The steps of adding a recognizable protection mark to the key parameter in step S23 are as follows:

[0072] S231. According to the functional importance of the parameters in the teacher model, the key parameters that affect the transfer of core knowledge in the teacher model are screened out, multiple parameter files are generated (the parameter file is a structured data set that stores the key parameters (such as weights and biases) in the teacher model, which are usually saved in groups by network layers or functional modules and used to identify and transfer core knowledge), and a unique identifier is assigned to each key parameter in each parameter file;

[0073] S232. Embed the identifier into the description field of the parameter file so that the identifier corresponds one-to-one with the key parameter values in the parameter file, forming a tag. Then generate a verification code for the tagged parameter file (the verification code generates a unique digest for the content of the tagged parameter file through a hashing algorithm, or generates a digital signature by combining key encryption to ensure data integrity and reliable source, and is independently stored in the security verification module for subsequent comparison), and store the verification code independently in the preset security verification module;

[0074] The tags of the key parameters in the parameter file are divided into: parameters with high-importance tags, parameters with medium-importance tags, and parameters with low-importance tags;

[0075] The specific steps for adding controllable interference to the key knowledge fragments in the teacher model according to the protection tag in step S23 are as follows:

[0076] S233. For the parameters with high-importance tags, superimpose random noise and limit the noise intensity not to exceed 15% of the original parameter value. For the parameters with medium-importance tags, perform interval translation or proportional scaling on the parameter values, and control the offset amplitude within 5% - 10%. For the parameters with low-importance tags, retain the original numerical value and only add a small amount of perturbation. For the parameters marked with high importance, generate a random noise value that follows a uniform distribution, and limit the absolute value range of the noise value to 0% to 15% of the original parameter value, and add the noise value and the original parameter value bit by bit to generate the perturbed parameter value. For the parameters marked with medium importance, perform any of the following operations: Interval translation: Generate a fixed offset, and the absolute value range of the offset is 5% to 10% of the original parameter value, and add the offset and the original parameter value. Proportional scaling: Generate a scaling factor, and the coefficient range is 0.95 to 1.05 (corresponding to -5% to +5% of the original value) or 0.90 to 1.10 (corresponding to -10% to +10% of the original value), and multiply the coefficient and the original parameter value. For the parameters marked with low importance, generate a random perturbation value with an absolute value not exceeding 1% of the original parameter value and superimpose it on the original parameter value;

[0077] S234. Generate an interference factor based on the hardware characteristics of the authorized user's device, so that the same key parameter produces a differentiated interference effect when running on different devices, and the interference factor is automatically updated every preset period.

[0078] Example 2. A certain school uses the teacher model to generate a lightweight student model for student teaching. In step S23:

[0079] Parameter tagging: Screen out the weights of the fully connected layer (high importance), the parameters of the feature fusion layer (medium importance), and the parameters of the preprocessing layer (low importance) that affect lesion localization, assign identifiers respectively, and generate verification codes;

[0080] Interference addition: Add 12% Gaussian noise to high-importance parameters; scale medium-importance parameters by 8%; add 0.5% minor perturbation to low-importance parameters;

[0081] Device binding: Generate interference factors according to the GPU model of the teacher terminal. When the same model runs on device A and device B, the parameter perturbation modes are different.

[0082] Through dynamic perturbation of parameter importance grading interference and device feature binding, while retaining the diagnostic accuracy of the model, attackers cannot restore the original teacher model through reverse engineering; the differential interference mechanism increases the difficulty of model stealing and reproduction, and ensures the integrity of key parameters through independent storage of verification codes, adapting to lightweight model deployment in high-security demand scenarios such as schools, medical care, and security.

[0083] S3. Knowledge transfer security protection: Train the lightweight student model, and verify the data access rights in real time during the training process, and dynamically adjust the protection intensity according to the training stage;

[0084] Among them, the specific content of the knowledge transfer security protection in step S3 includes:

[0085] S31. Set security constraint rules for data input and model weight update during the training process according to the key parameters and protection marks in the lightweight student model, and bind the preset permission verification interface;

[0086] S31. Security constraint rule setting and permission binding, including the following steps:

[0087] S31a. Definition of key parameter constraint rules: Extract parameters marked as high, medium, and low importance from the lightweight student model, and set the following rules according to the parameter importance level: High-importance parameters: It is prohibited to directly modify the numerical value during the training process, and only parameter updates are allowed through the preset encrypted channel; Medium-importance parameters: Limit the amplitude of numerical change in a single training iteration not to exceed 10% of the original value; Low-importance parameters: Allow free adjustment, but the operator and timestamp of each update need to be recorded;

[0088] S31b. Permission verification interface binding: Associate the security constraint rules with the identity identification of the authorized device and the user access permission level to form a permission verification mapping table, and embed it in the initialization configuration of the training process;

[0089] Method for generating the permission verification mapping table: Map the user role (such as administrator, ordinary user), device ID (MAC address or digital certificate) and parameter constraint rules (high / medium / low importance), and generate a key-value pair table through the policy engine;

[0090] The content in the permission verification mapping table includes: device ID, user permission level, parameter types allowed for operations (high / medium / low), data access scope, and operation validity period;

[0091] S32. Divide the training process into multiple training cycles. Before the start of each training cycle, call the permission verification interface to verify the data access permission of the authorized user's device. If the permission is invalid, suspend the training. If the verification passes, load the corresponding training data;

[0092] The verification process for data access permission is as follows: Before training, send the device ID and user token to the security verification module. Verify the legality by comparing the permission level, data sensitivity (such as public / confidential), and the current training stage (such as pre-training / fine-tuning) in the mapping table. If the token has expired or the permissions do not match, reject the data loading;

[0093] S33. Dynamically adjust the protection intensity for the weight update operation of the lightweight student model according to the training cycle and the sensitivity level of the training data, including adding an encryption verification layer for increasing the gradient update frequency and restricting the input scale of single training data.

[0094] S33. Dynamic protection intensity adjustment includes the following steps:

[0095] S33a. Training stage and data sensitivity classification: Divide the training process into three stages: initial stage, middle stage, and late stage, and label the sensitivity level (high, medium, low) for the training data;

[0096] Initial stage (the first 20% of the cycles): Set the protection intensity to the baseline level and allow the maximum data input scale;

[0097] Middle stage (21% - 80% of the cycles): If the data sensitivity is high, reduce the single input data volume by 50% and increase the encryption verification frequency during parameter update;

[0098] Late stage (81% - 100% of the cycles): For high and medium sensitivity data, increase the encryption verification frequency to 3 times per cycle, and limit the single input data volume to 30% of the initial stage;

[0099] S33b. Protection intensity execution and feedback: During the training process, if an unauthorized operation or data anomaly is detected, immediately raise the protection intensity to the highest level (terminate data input and freeze parameter update). After the training is completed, generate an adjustment report based on the protection log to optimize the protection strategy for the subsequent training stage;

[0100] During the training process, monitor the abnormal fluctuations of the model parameters or unauthorized data call requests in real time. If an abnormal operation is detected, terminate the training process and generate a security log containing the abnormal type, timestamp, and operation source.

[0101] Multi-level security protection for the training process is achieved through parameter importance classification constraints and dynamic binding of permissions. Combined with the linkage adjustment of the training phase and data sensitivity, efficiency and risk are dynamically balanced while ensuring model accuracy. Real-time monitoring and exception blocking mechanisms effectively prevent data leakage and malicious tampering, and subsequent strategies are optimized through closed-loop management of protection logs to ensure the controllability and compliance of the entire lightweight model migration process.

[0102] S4. Deployment environment security binding, bind the trained lightweight student model with the preset target running device for verification, and deploy a real-time behavior monitoring module to monitor abnormal operations of the lightweight student model during loading and reasoning;

[0103] The specific steps of deploying environment security binding in step S4 include:

[0104] S41, bind the trained lightweight student model to the target running device for verification. If the target running device is a device of an authorized user, no verification is required, and the behavior monitoring module is used to monitor abnormal operations of the lightweight student model during loading and reasoning. If the target running device is not a device of an authorized user, jump to step S42;

[0105] S42. The authorized user's device sends a temporary verification key containing a timestamp to the target running device, and requires the device identity matching to be completed through the authorized user's authentication information within the preset time. If the target running device fails to complete the verification within the preset time, the lightweight student model will be transferred to an isolated storage area, and a security alarm log containing the device identification of the target running device, the reason for the verification failure and the operation time will be generated. If the verification is completed, the behavior monitoring module will be used to monitor abnormal operations of the lightweight student model during the loading and reasoning process.

[0106] The specific steps of monitoring abnormal operation of the lightweight student model during loading and reasoning in step S41 are as follows:

[0107] S411, when the lightweight student model is loaded into the target running device, the hash value of the model file of the lightweight student model is compared with the pre-stored standard hash value in real time. If they are inconsistent, it is determined that the model has been tampered with, triggering a loading interrupt and generating a tampering alarm. If they are consistent, continue to compare in real time;

[0108] S412. During the lightweight student model reasoning process, the format, value range and source of the input data are verified in real time. If the verification detects a data input request that exceeds a preset threshold or is unauthorized, the data is restricted from entering the reasoning process and marked as abnormal input;

[0109] S413. When the behavior monitoring module detects tampering, abnormal input, or unauthorized model call behavior during the loading or inference process, it immediately terminates the current operation, freezes the running state of the lightweight student model, and generates a security log containing the type of anomaly, trigger time, and operation context, which is synchronously uploaded to the audit management module.

[0110] Embodiment 3. An AI-assisted diagnosis system in a certain top-three hospital deploys a lightweight student model for CT image analysis;

[0111] Deployment verification: The target device is a radiology department terminal (device ID: CT-ROOM-01), and the model is loaded after successful binding verification;

[0112] Anomaly detection: Loading stage: During a certain model loading, the hash value comparison fails, triggering a tampering alarm, and the model is transferred to the isolation area;

[0113] Inference stage: An external device attempts to input non-standard DICOM data (resolution exceeding the limit), which is marked as abnormal input and intercepted;

[0114] Log generation: The security log records the tampering alarm and abnormal input, and is synchronously uploaded to the audit module for subsequent traceability.

[0115] Through device binding verification and multi-level abnormal operation monitoring, it is ensured that the lightweight student model runs securely only on authorized devices, preventing model tampering or malicious calls; real-time hash verification and compliance interception of input data block abnormal inference operations, ensuring the reliability of model inference results; the linkage between the security log and the audit module realizes full-process traceability of the deployment, meeting the compliance requirements of highly sensitive scenarios such as medical and financial fields.

[0116] S5. Full-cycle audit tracking, establish a traceable audit management module, record the full-process operation logs from teacher model compression to the operation of the deployed lightweight student model, generate non-tamperable verification marks for the key operation nodes in the full-process operation logs, and regularly generate a security assessment report.

[0117] The full-process operation log refers to the structured data that completely records all operations (such as parameter modification, permission verification, model update) during the process of teacher model compression, student model generation, training, and deployment operation. It includes the operation type, executor, timestamp, details of parameter changes, and verification marks, and is used for tracing security events and compliance audits.

[0118] Among them, the key operation nodes in step S5 include teacher model compression, lightweight student model training, and deployment operation;

[0119] In step S5, generating non-tamperable verification marks for the key operation nodes in the full-process operation logs specifically includes:

[0120] S51. For the key operation nodes in the full-process operation log, extract the operation content, timestamp, and operator identity information, generate the corresponding encrypted digest through a preset one-way hashing function, and bind and store the encrypted digest with the node data;

[0121] The specific steps for generating the encrypted digest through the one-way hashing function and binding and storing it with the node data are as follows:

[0122] S51a. Preprocess the node data, extract the operation content, timestamp, and operator identity information of the key operation nodes, and serialize them into a standardized data block according to preset rules to ensure data format consistency;

[0123] S51b. Generate the one-way encrypted digest. Input the standardized data block into a preset one-way encryption algorithm to generate a unique identification code of a fixed length. This identification code corresponds one-to-one with the original data and cannot be reversely deduced;

[0124] S51c. Bind and store the digest with the data. Embed the encrypted digest as an independent field into the corresponding node data record, and at the same time store the algorithm version, timestamp, and operator identity identifier used when generating the digest to form a verifiable log entry containing complete metadata;

[0125] S51d. Synchronously store the verification parameters. Store the algorithm parameters (such as salt value, number of iterations) for generating the encrypted digest separately from the node data into a secure configuration library to ensure that the digest calculation result can be reproduced with the same parameters during subsequent verification;

[0126] S52. Concatenate the encrypted digests of each key operation node according to the operation time sequence. Each time a new key operation node is added, use the encrypted digest of the previous key operation node as an input parameter to participate in the hashing calculation of the current node, forming a chain-linked verification tag sequence to ensure that any tampering of node data will cause the verification tags of all subsequent nodes to become invalid;

[0127] Serialize the operation content, timestamp, and operator information of the current key operation node into a standardized data block, and splice it with the encrypted digest of the previous node in a preset order to form the complete input data to be calculated; Input the spliced input data into a preset irreversible encryption algorithm to generate a unique encrypted digest of a fixed length; Use the generated encrypted digest as one of the input parameters for the next node to ensure a one-way dependency relationship between nodes. Any tampering of node data will cause the digests of all subsequent nodes to become invalid; When generating the encrypted digest, add a unique random identifier (such as an operation serial number or device fingerprint) to prevent the risk of conflicts in generating the same digest by different nodes.

[0128] The above are only embodiments of the present invention, and do not thus limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, shall similarly be included within the patent protection scope of the present invention.

Claims

1. A lightweight model transfer learning method based on knowledge distillation, characterized in that: Including the following steps: S1. Construction of a security initialization module. Deploy a teacher model on a preset cloud server, and establish a dynamic permission verification mechanism. After authorizing a user to pass through the dynamic permission verification mechanism, the user can call the teacher model through a preset device and transmit it; S2. Generation of a lightweight student model. A preset processing module receives the teacher model, performs model compression processing on the teacher model, adds protection marks to the key parameters generated during the compression process, and adds controllable interference to the key knowledge fragments in the teacher model according to the protection marks to generate a lightweight student model; The steps for generating the lightweight student model in step S2 include: S21. Perform hierarchical analysis on the network structure of the teacher model through the processing module, identify and remove duplicate or redundant components, and generate a simplified basic model framework; S22. In the basic model framework, screen and merge the parameter modules of the teacher model with similar functions, reduce the overall number of parameters of the teacher model, and retain the key parameters required for the core knowledge transfer in the teacher model; S23. Add recognizable protection marks to the key parameters, and add controllable interference to the key knowledge fragments in the teacher model according to the protection marks, and finally generate a lightweight student model; The steps for adding recognizable protection marks to the key parameters in step S23 are as follows: S231. According to the functional importance of the parameters in the teacher model, screen out the key parameters that affect the core knowledge transfer in the teacher model, generate multiple parameter files, and assign a unique identifier to each key parameter in each parameter file; S232. Embed the identifier into the description field of the parameter file, so that the identifier corresponds one-to-one with the key parameter value in the parameter file to form a mark, and generate a verification code for the marked parameter file, and store the verification code independently in a preset security verification module; The marking of the key parameters in the parameter file is divided into: parameters marked with high importance, parameters marked with medium importance, and parameters marked with low importance; The specific steps for adding controllable interference to the key knowledge fragments in the teacher model according to the protection marks in step S23 are as follows: S233. For the parameters marked with high importance, superimpose random noise and limit the noise intensity not to exceed 15% of the original parameter value. For the parameters marked with medium importance, perform interval translation or proportional scaling on the parameter value, and control the offset range at 5% - 10%. For the parameters with low importance, retain the original value and only add a small amount of perturbation; S234. Generate an interference factor according to the hardware characteristics of the device of the authorized user, and the interference factor is automatically updated every preset period; S3. Knowledge transfer security protection. Train the lightweight student model, and verify the data access permission in real time during the training process, and dynamically adjust the protection intensity according to the training stage; S4. Deploy the environmental security binding, bind and verify the trained lightweight student model with a preset target running device, and deploy a real-time behavior monitoring module to monitor abnormal operations of the lightweight student model during loading and inference; S5. Conduct full-cycle audit and tracking, establish a traceable audit management module, record the full-process operation logs from teacher model compression to the operation of the deployed lightweight student model, generate non-tamperable verification marks for key operation nodes in the full-process operation logs, and regularly generate security assessment reports.

2. The lightweight model transfer learning method based on knowledge distillation according to claim 1, wherein: The dynamic permission verification mechanism in step S1 includes the following content: S11. When the cloud server receives a device call request from an authorized user, it detects whether the IP address, GPS location, and device ID of the authorized user's device are within the pre-authorized whitelist through a preset verification unit; S12. If the verification passes, the cloud server generates a one-time dynamic token, matches the one-time dynamic token with the fingerprint information entered in the authorized user's device, and transmits the one-time dynamic token to the authorized user's device after successful matching; S13. After receiving the one-time dynamic token, the authorized user's device conducts local verification to ensure that the one-time dynamic token matches the fingerprint information entered in the authorized user's device correctly, establishes a connection with the cloud server through a secure channel, and encrypts and transmits the teacher model to the processing module.

3. The lightweight model transfer learning method based on knowledge distillation according to claim 2, characterized in that: When the authorized user calls the teacher model in step S1, the permission level is adjusted according to the authorized user behavior score. If the score is lower than the preset threshold, the permission level is lowered and the model access frequency is restricted; if the score is higher than the threshold, the permission level is raised and the model access permission is increased, and a one-time dynamic token is regenerated after the permission change to ensure the uniqueness of the one-time dynamic token for each permission change and prevent unauthorized access; If the score is equal to the threshold, the permission level remains unchanged.

4. The lightweight model transfer learning method based on knowledge distillation according to claim 3, characterized in that: The operation permission levels of the authorized user are divided into: model query, parameter reading, and full call; The behavior scores of the authorized user include: authentication behavior score, operation behavior score, device environment change score, and data interaction behavior score.

5. The lightweight model transfer learning method based on knowledge distillation according to claim 4, wherein: The specific content of knowledge transfer security protection in step S3 includes: S31. According to the key parameters and protection marks in the lightweight student model, set security constraint rules for data input and model weight update during the training process, and bind a preset permission verification interface; S32. Divide the training process into multiple training cycles. Before the start of each training cycle, call the permission verification interface to verify the data access permission of the authorized user's device. If the permission is invalid, the training is suspended; if the verification passes, the corresponding training data is loaded; S33. Dynamically adjust the protection intensity of the lightweight student model weight update operation according to the training cycle and the sensitivity level of the training data, including adding an encryption verification layer for increasing the gradient update frequency and restricting the input scale of single training data.

6. The lightweight model transfer learning method based on knowledge distillation according to claim 5, characterized in that: The specific steps of deploying environmental security binding in step S4 are as follows: S41. Bind and verify the trained lightweight student model with the target operating device. If the target operating device is the device of an authorized user, verification is not required, and the behavior monitoring module is used to monitor abnormal operations during the loading and inference processes of the lightweight student model. If the target operating device is not the device of an authorized user, jump to step S42; S42. The device of the authorized user sends a temporary verification key containing a timestamp to the target operating device and requests to complete device identity matching through the authentication information of the authorized user within a preset time. If the target operating device fails to complete the verification within the preset time, transfer the lightweight student model to an isolated storage area and generate a security warning log containing the device identifier of the target operating device, the reason for verification failure, and the operation time. If the verification is completed, use the behavior monitoring module to monitor abnormal operations during the loading and inference processes of the lightweight student model.

7. The lightweight model transfer learning method based on knowledge distillation according to claim 6, characterized in that: The specific steps of monitoring abnormal operations during the loading and inference processes of the lightweight student model in step S41 are as follows: S411. When the lightweight student model is loaded onto the target operating device, compare the hash value of the model file of the lightweight student model with the pre-stored standard hash value in real time. If they are inconsistent, it is determined that the model has been tampered with, trigger a loading interruption and generate a tampering warning. If they are consistent, continue the real-time comparison; S412. During the inference process of the lightweight student model, perform real-time verification on the format, numerical range, and source of the input data. If it is detected during verification that the data exceeds the preset threshold or an unauthorized data input request, restrict the data from entering the inference process and mark it as abnormal input; S413. When the behavior monitoring module detects tampering, abnormal input, or unauthorized model call behavior during the loading or inference process, immediately terminate the current operation, freeze the running state of the lightweight student model, and generate a security log containing the abnormal type, trigger time, and operation context, and synchronously upload it to the audit management module.

8. The lightweight model transfer learning method based on knowledge distillation according to claim 7, wherein: The key operation nodes in step S5 include teacher model compression, lightweight student model training, and deployment operations; Generating an immutable verification mark for the key operation nodes in the full-process operation log in step S5 specifically includes: S51. For the key operation nodes in the full-process operation log, extract the operation content, timestamp, and operator identity information, generate a corresponding encrypted digest through a preset one-way hash function, and bind and store the encrypted digest with the node data; S52. Concatenate the encrypted digests of each key operation node in the order of operation timing. Each time a new key operation node is added, use the encrypted digest of the previous key operation node as an input parameter to participate in the hash calculation of the current node to form a chain-linked verification mark sequence.

Images