Adaptive model transfer method and system based on reverse multi-professor distillation

Abstract

This specification relates to several embodiments, specifically to an adaptive model transfer method and system based on reverse multi-professor distillation. The method includes the following steps: acquiring parameters of multiple target devices that may be transferred, and obtaining a reference device with the lowest computing power; acquiring multiple reference scenarios and corresponding sample data, assigning a reference device to each reference scenario, and establishing a model; training the corresponding model using the corresponding sample data, denoted as the reference scenario model; establishing a student model, using a pre-accessed large model to guide the training of the student model, serving as the standard model; acquiring a selected target application scenario, and obtaining the correlation between the target application scenario and each of the reference scenarios; selecting multiple reference scenario models with correlation higher than a preset reference threshold as professor models; using the multiple professor models to teach the standard model, and after completing the teaching, transferring the standard model to the target device.

Description

Technical Field

[0001] This specification relates to several embodiments, and specifically to an adaptive model transfer method and system based on reverse multi-professor distillation. Background Technology

[0002] With the rapid development of artificial intelligence technology, deep learning models have achieved remarkable results in many fields such as image recognition, natural language processing, and speech recognition. However, these high-performance models typically require a large amount of computing resources and storage space, which limits their application on edge devices, mobile terminals, or low-computing-power devices. To overcome this limitation, model compression and transfer learning techniques have gradually gained attention. In particular, knowledge distillation methods, by transferring knowledge from a large teacher model to a small student model, enable lightweight models to reduce the demand for computing resources while maintaining high accuracy.

[0003] However, in practical applications, the hardware differences between different target devices and the diversity of application scenarios bring new challenges to model transfer. Traditional knowledge distillation guided by a single teacher model is difficult to adapt to the complex cross-device and cross-scenario transfer requirements. Therefore, it is necessary to study model miniaturization and techniques to further enhance generalization. Summary of the Invention

[0004] This specification describes an adaptive model transfer method and system based on reverse multi-professional distillation through several embodiments.

[0005] Firstly, the embodiments of this specification provide an adaptive model transfer method based on reverse multi-professor distillation, including the following steps:

[0006] Obtain parameters of multiple target devices that may be migrated, and obtain a reference device with the lowest computing power based on the parameters of the multiple target devices;

[0007] Acquire multiple reference scenarios and corresponding sample data for each reference scenario, assign a reference device to each reference scenario, and build a model on the reference device whose scale matches the parameters of the reference device;

[0008] Each reference scenario's sample data is used to train the corresponding model on the reference device, and the trained model is denoted as the reference scenario model.

[0009] Based on the parameters of the reference device, a student model is established, and a pre-accessed large model is used as a teacher model to guide the training of the student model. The trained student model is then used as a standard model.

[0010] Obtain the selected target application scenario and obtain the correlation between the target application scenario and each of the reference scenarios;

[0011] Multiple reference scenario models with a correlation degree higher than a preset reference threshold are selected as teaching models;

[0012] The standard model is taught using multiple of the aforementioned teaching models, and after the teaching is completed, the standard model is transferred to the target device.

[0013] Secondly, embodiments of this specification provide an adaptive model transfer system based on reverse multi-professor distillation, including:

[0014] The reference device module obtains parameters of multiple target devices that may be migrated, and obtains the reference device with the lowest computing power based on the parameters of the multiple target devices;

[0015] The reference scenario module acquires multiple reference scenarios and corresponding sample data, assigns a reference device to each reference scenario, and establishes a model on the reference device whose scale matches the parameters of the reference device.

[0016] The reference model module uses the sample data corresponding to each reference scenario to train the model on the corresponding reference device. The trained model is denoted as the reference scenario model.

[0017] The standard model module establishes a student model based on the parameters of the reference device, uses a pre-accessed large model as a teacher model to guide the training of the student model, and the trained student model serves as the standard model.

[0018] The association module obtains the selected target application scenario and the degree of association between the target application scenario and each of the reference scenarios;

[0019] The selection module selects multiple reference scenario models whose correlation degree is higher than a preset reference threshold as teaching models;

[0020] The migration module teaches the standard model using multiple teaching models, and after the teaching is completed, migrates the standard model to the target device.

[0021] Thirdly, embodiments of this specification provide an electronic device, including a processor and a memory;

[0022] The processor is connected to the memory;

[0023] The memory is used to store executable program code;

[0024] The processor runs a program corresponding to the executable program code stored in the memory to perform the method described in any of the above aspects.

[0025] Fourthly, embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the methods described in any of the above aspects.

[0026] Fifthly, embodiments of this specification provide a computer program product, including a computer program that, when executed by a processor, implements the methods described in any of the above aspects.

[0027] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0028] In several embodiments of this specification, the adaptive model transfer method and system based on reverse multi-teacher distillation dynamically selects multiple reference scenario models most suitable for the target application scenario as teacher models. This allows for the extraction of knowledge from different perspectives and its integration into the student model, improving not only the model's adaptability to specific tasks but also its generalization ability in new scenarios. A reference device with the lowest computing power is generated based on the specific parameters of the target device, and a matching model is built upon this, ensuring efficient operation of the model in resource-constrained environments. Traditional knowledge distillation guided by a single-teacher model struggles to handle complex cross-device and cross-scenario transfer requirements. The reverse multi-teacher distillation strategy employed in this application, combined with correlation-based teacher model selection, effectively improves the efficiency and generalization ability of the transferred model.

[0029] Other features and advantages of various embodiments of this specification will be further revealed in the following detailed description and accompanying drawings. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram of adaptive model migration provided for an embodiment of this specification.

[0032] Figure 2 This is a schematic diagram of the adaptive model transfer method provided in the embodiments of this specification.

[0033] Figure 3 This diagram illustrates the acquisition of correlation degree as provided in the embodiments of this specification.

[0034] Figure 4 This is a schematic diagram of an adaptive model transfer system provided in the embodiments of this specification.

[0035] Figure 5A schematic diagram of an electronic device provided in an embodiment of this specification. Detailed Implementation

[0036] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of this specification and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of this specification.

[0037] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0038] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to facilitate the description of the embodiments and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this specification.

[0039] All data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0040] Before introducing the technical solutions described in this manual, the application scenarios and related technologies of the technical solutions will be introduced.

[0041] Model transfer is a technique that allows a machine learning model trained on one environment or device to be transferred and effectively run on another different environment or device. This technique is particularly suitable for situations where the target device has limited resources, such as mobile devices and embedded systems. Through model transfer, complex models that originally required powerful computing capabilities can be adapted to devices with lower computing power, thereby expanding the application scope of the model.

[0042] The significance of model migration lies in breaking the limitations of high-performance computing resources, enabling advanced machine learning models to be used under various hardware conditions. Especially in fields such as the Internet of Things (IoT) and edge computing, directly deploying large-scale models is often impractical due to the diversity of devices and their limited computing resources. Model migration provides an effective solution that ensures model performance while complying with device resource constraints.

[0043] For example, in smart home systems, smart speakers from different brands have different hardware configurations. Suppose we develop a complex speech recognition model designed to improve the understanding of specific sentences. However, high-end smart speakers are equipped with powerful processors and ample memory, while low-end models are relatively limited. Using the adaptive model transfer method based on reverse multi-teacher distillation described above, we can adjust and optimize this complex speech recognition model so that it can not only run efficiently on high-end devices but also adapt to the resource limitations of low-end devices, ensuring that all users can experience high-quality voice interaction services. In this way, even with differences in hardware conditions, all smart speakers can maintain a consistent service quality and user experience.

[0044] This specification provides an adaptive model transfer method and system based on reverse multi-professor distillation. Please refer to the appendix. Figure 1 This method and system first selects a reference device 11 with the lowest computing power based on the parameters of multiple potential target devices 24, and establishes a scale-matched model on this device. Then, it trains the corresponding model using sample data from each reference scenario 13. Next, a pre-accessed large model is used as a teacher model to guide the training of student models, ultimately obtaining a standard model 22. By analyzing the correlation between the target application scenario 21 and the reference scenario 13, an appropriate reference scenario model 12 is selected as the teaching model to instruct the standard model 22. Finally, the optimized standard model 22 is migrated to the target device 24.

[0045] Specifically, this specification first provides an adaptive model transfer method based on reverse multi-professor distillation; please refer to the appendix. Figure 2 The steps include:

[0046] Step S1) Obtain parameters of multiple target devices 24 that may be migrated, and obtain a reference device 11 with the lowest computing power based on the parameters of the multiple target devices 24. Specifically, this includes:

[0047] The hardware parameter information of the target device 24 is collected. The hardware parameters include processor type, number of cores, memory capacity, available storage space, floating-point operation capability, maximum power consumption limit and supported inference framework.

[0048] The performance of multiple target devices 24 is normalized, and the hardware parameters of different dimensions are unified into performance indicators to obtain the comprehensive computing power evaluation value of the target devices 24.

[0049] Based on the comprehensive computing power evaluation value of all target devices 24, the target device 24 with the lowest comprehensive computing power evaluation value is selected as the reference device 11.

[0050] For example, in this embodiment, an intelligent image recognition system needs to be deployed on three different devices. Hardware parameter information of the target device 24 is collected to obtain:

[0051] Device A: Processor type: Intel Xeon E5-2699 v4, Number of cores: 22, Memory capacity: 128GB, Available storage space: 2TB SSD, Floating-point performance: 3.8 TFLOPS, Maximum power limit: 145W, Supported inference frameworks: TensorFlow, PyTorch.

[0052] Device B: Processor type: Intel Core i7-9700K, Number of cores: 8, Memory capacity: 16GB, Available storage space: 512GB SSD, Floating-point performance: 1.5 TFLOPS, Maximum power limit: 95W, Supported inference frameworks: TensorFlow, ONNX Runtime.

[0053] Device C: Processor type: Qualcomm Snapdragon 855, Number of cores: 8, Memory capacity: 6GB, Available storage space: 128GB UFS, Floating-point performance: 0.7 TFLOPS, Maximum power limit: 5W, Supported inference framework: TensorFlow Lite.

[0054] Hardware parameters are normalized for performance to facilitate comparison. A method disclosed in the art is used to convert each parameter into a relative value or score, and then weighted summation to obtain a comprehensive computing power evaluation value. For example, weights are assigned based on factors such as processor performance, memory size, and floating-point arithmetic capability, and a score representing the overall performance of the device is calculated based on these weights.

[0055] For example, floating-point arithmetic capability is used as the primary performance indicator with the highest weight, while other hardware parameters have lower weights. The final calculated score is 3.8 for Device A, 1.5 for Device B, and 0.7 for Device C. Therefore, Device C is selected as reference device 11. This means the final migrated model needs to run smoothly on Device C.

[0056] Step S2) Obtain multiple reference scenarios 13 and corresponding sample data for each reference scenario 13, assign a reference device 11 to each reference scenario 13, and establish a model on the reference device 11 whose scale matches the parameters of the reference device 11.

[0057] Different reference scenarios 13 are defined or identified, each with its specific application background and requirements, as well as a corresponding sample dataset. For example, in the field of image recognition, reference scenarios 13 include face recognition, object detection, and text recognition.

[0058] Based on the selected reference device 11 with the lowest computing power, one reference device 11 is allocated to each reference scenario 13. The hardware parameters of the reference devices 11 are the same, only the instances are different. That is, there are multiple reference devices 11, but the hardware parameters are the same. An initial-scale model is initialized on the selected reference device 11. The model has a certain preset number of layers, number of neurons, and weight settings.

[0059] The step of establishing a model on the reference device 11 whose scale matches the parameters of the reference device 11 includes:

[0060] An initial model of a preset initial scale is established and run on the reference device 11, and the running time is recorded. The initial model has a preset number of layers, number of neurons, and weights.

[0061] When the running time is less than a preset lower reference time, the number of neurons is increased; when the running time is greater than a preset upper reference time, the number of neurons is decreased, until the running time is between the lower reference time and the upper reference time.

[0062] Specifically, run the model and record the runtime. If the model runtime is too short (below the preset lower reference time), it indicates that the model size may be too small and cannot fully utilize device resources. In this case, consider increasing the number of neurons to improve model complexity. Conversely, if the runtime is too long (exceeding the preset upper reference time), it may mean that the model is too complex and unsuitable for the current device. In this case, the number of neurons should be reduced to optimize the model size. Continuously adjust the model size until its runtime is within a reasonable range.

[0063] Step S3) Use the sample data corresponding to each reference scenario 13 to train the model on the corresponding reference device 11. The trained model is denoted as reference scenario model 12.

[0064] Each reference scenario 13 requires a corresponding sample dataset. The dataset should be representative of various situations within that scenario and possess sufficient diversity to train a robust model. Based on the model size (number of layers, number of neurons, etc.) determined in step S2, the model is configured on the reference device 11, including loading initial weights and setting hyperparameters such as the learning rate. The model is trained using the sample dataset corresponding to reference scenario 13. Data is input into the model, and the model parameters are adjusted based on the difference between the model output and the actual labels.

[0065] During training, various techniques can be employed to improve model performance and training efficiency, such as batch gradient descent, stochastic gradient descent (SGD), and the Adam optimizer. Key metrics during training are monitored to evaluate the model's learning progress and effectiveness. After training, the model is evaluated using a validation set to check its generalization ability. Once the model reaches a satisfactory performance level, it is saved as reference scenario model 12. Reference scenario model 12 is specifically optimized for a particular reference scenario 13, providing relatively good performance in the same or similar application context.

[0066] For example, referring to scenario 13 "Basic Object Classification", reference device 11 is Device C. Photos containing everyday objects are collected, covering different categories such as books, cups, and mobile phones. Due to the hardware limitations of Device C, a lightweight CNN model is deployed on it. The model is trained using the aforementioned sample dataset, monitoring the loss function's decreasing trend and classification accuracy during the process. The model's performance is checked using cross-validation, and the model architecture or training parameters are adjusted as necessary. Once the model demonstrates good classification ability and adapts to the resource limitations of Device C, it is saved as reference scenario model 12 for subsequent knowledge distillation tasks.

[0067] Step S4) Based on the parameters of the reference device 11, establish a student model, use the pre-accessed large model as the teacher model to guide the training of the student model, and use the trained student model as the standard model 22.

[0068] Based on the hardware parameters of reference device 11 (such as processor type, number of cores, memory capacity, etc.), a student model suitable for running on this device is designed. This typically means choosing a smaller model architecture to accommodate lower computing resources and storage space. The design of the student model should consider that its complexity matches the performance of reference device 11, ensuring that the model not only runs efficiently on the device but also achieves the expected accuracy.

[0069] The teacher model is a pre-trained model with high accuracy, providing rich feature representations and predictive capabilities. During training, the teacher model's output (e.g., predicted probability distributions, intermediate layer feature representations, etc.) is used to guide the student model's learning. This method is called "knowledge distillation," enabling the student model to learn knowledge from the teacher model, even if the student model is much smaller than the teacher model.

[0070] During training, in addition to traditional loss functions (such as cross-entropy loss, which measures the accuracy of the student model's predictions on the sample data), additional loss terms are introduced, such as defining the loss function by comparing the output differences between the student model and the teacher model, to improve the performance of the student model.

[0071] After training, if the student model can effectively learn from the teacher model and shows good performance on the validation set, then it can be identified as the standard model 22. The standard model 22 retains the essential knowledge of the teacher model and has the ability to adapt to low-resource environments, making it suitable for the target application scenario 21.

[0072] Step S5) Obtain the selected target application scenario 21 and obtain the correlation degree between the target application scenario 21 and each of the reference scenarios 13. Specifically, this includes:

[0073] Extract the feature description information of the target application scenario 21, denoted as the target feature description information. The feature description information includes the input data modality, output index, data distribution characteristics, target device 24 type, and application scenario description.

[0074] Extract the feature description information for each reference scene 13, and denote it as the reference feature description information;

[0075] The target feature description information is compared with the reference feature description information to obtain the correlation between the target application scenario 21 and each reference scenario 13.

[0076] The feature description information of target application scenario 21 (target feature description information) includes input data modality, output indicators, data distribution characteristics, target device type 24 and application scenario description.

[0077] The input data modality is used to determine the data type used in the target application scenario 21, such as images, audio, and text. The output metrics are used to define the expected type of result from this application scenario, such as classification labels or numerical predictions. Data distribution characteristics are used to analyze and record the statistical properties of the input dataset, including but not limited to mean, variance, and data distribution shape. The target device type 24 is used to identify the hardware platform planned for deployment in the target application scenario 21, taking into account factors such as computing power and memory size. The application scenario description details the purpose and usage scenarios of the application, helping to understand its functional and performance requirements.

[0078] For each reference scenario 13, the feature description information of the above five aspects is also extracted, so that it can be directly compared with the target scenario. Semantic similarity: The semantic similarity between the description of the target application scenario 21 and the description of the reference scenario 13 is analyzed using natural language processing techniques. This helps us understand the similarity in purpose and use between the two. Data distribution similarity: Based on the statistical attributes of the input data, such as mean and variance, the similarity of the data distribution between the target scenario and the reference scenario 13 is calculated. This is crucial for ensuring the model's generalization ability on different datasets. Task structure similarity: Based on information such as input data modality and output metrics, the similarity of the task structures of the two scenarios is evaluated. For example, if both scenarios are based on image data for object classification, their task structure similarity is high. Comprehensive correlation calculation: Combining the above three dimensions (semantic similarity, data distribution similarity, and task structure similarity), a weighted summation or other appropriate method is used to calculate the overall correlation between the target application scenario 21 and each reference scenario 13. The weights can be adjusted according to the specific application background and requirements.

[0079] For example, the application scenario is "urban traffic flow monitoring," requiring the identification of the most suitable reference scenario 13 for model transfer. Target feature description information includes input data modality as video streams (frame images) and output metric as vehicle quantity estimates. Data distribution characteristics include significant day / night illumination variations and diverse weather conditions. Target device 24 is an edge computing device with limited GPU resources. The application scenario is described as real-time monitoring of traffic flow at major road intersections to support intelligent traffic management decisions.

[0080] The following reference scenario 13 and reference scenario model 12 are provided. Reference scenario 13A is "parking space detection". Input data modality: static images captured by a fixed camera. Output index: number of available parking spaces. Data distribution characteristics: stable indoor environment with little change in lighting. Reference device 11 type: ordinary PC. Application scenario description: assisting the parking management system in automatically updating parking space status.

[0081] The correlation between "urban traffic flow monitoring" and "parking lot space detection" may not be very high because, although both involve vehicle detection, there are significant differences in input data modalities, data distribution characteristics, and application scenario descriptions. Therefore, we will continue to look for other closer reference scenarios, such as "vehicle recognition at highway entrances and exits," whose feature description information is more consistent with the target scenario, thereby improving the success rate of model transfer.

[0082] This embodiment can assess the similarity or correlation between the target application scenario 21 and each reference scenario 13, thereby providing a basis for the subsequent selection of the most suitable target model 23 for migration.

[0083] Please see the appendix Figure 3 The steps of comparing the target feature description information with the reference feature description information to obtain the correlation between the target application scenario 21 and each reference scenario 13 include:

[0084] Semantic similarity is calculated by identifying the semantics of the application scenario descriptions in the target feature description information and the reference feature description information, respectively.

[0085] Based on the input data modalities and data distribution characteristics of the target feature description information and the reference feature description information, the data distribution similarity is obtained;

[0086] The task structure similarity is obtained based on the input data modality, output index, and data distribution characteristics of the target feature description information and the reference feature description information;

[0087] The correlation between the target application scenario 21 and each reference scenario 13 is obtained by weighting the semantic similarity, data distribution similarity and task structure similarity.

[0088] To assess the correlation between the target application scenario 21 and each reference scenario 13, a multi-dimensional comparison was conducted, including semantic similarity, data distribution similarity, and task structure similarity.

[0089] The application scenario description is extracted from the target feature description information and the reference feature description information. Using word vector models (such as Word2Vec, GloVe) or sentence embedding models (such as BERT), the application scenario description is transformed into a numerical vector form, and the semantic similarity between these vectors is measured by calculating the distance between them (such as cosine similarity).

[0090] Compare the data types (e.g., images, audio, etc.) and statistical properties (e.g., mean, variance) of the target scene and the reference scene 13. For example, in an image classification task, the color histogram, texture features, etc., of the images can be analyzed. Use statistical tests (e.g., the Kolmogorov-Smirnov test) or feature-representation-based distance metrics (e.g., Euclidean distance, Mahalanobis distance) to quantify the similarity of distributions between different datasets.

[0091] In addition to considering the nature of the data itself, it is also necessary to pay attention to the specific requirements of the task, such as what type of signal the input is, what kind of output result is expected (classification, regression, etc.), and how to handle different types of data distributions. Calculating task structural similarity involves defining a series of rules or using machine learning methods to automatically evaluate the structural similarity between two tasks. For example, if two tasks are both image classification and have a similar number of categories, then their task structural similarity is considered high.

[0092] Based on specific application requirements, different weights are assigned to semantic similarity, data distribution similarity, and task structure similarity. These weights are determined through learning from historical data. The three similarities are then summed according to their respective weights to obtain the final relevance score between the target application scenario 21 and each reference scenario 13.

[0093] For example, target application scenario 21 is "urban traffic flow monitoring," while a reference scenario 13 is "vehicle recognition at highway entrances and exits." Regarding semantic similarity, both involve vehicle detection, but one focuses on traffic flow monitoring, while the other focuses on vehicle entry and exit management. Semantic similarity scores are calculated between the two using semantic analysis tools. Regarding data distribution similarity, considering that both process vehicle images in a road environment, although lighting conditions and shooting angles may differ, they still exhibit high data distribution similarity. Regarding task structure similarity, both inputs are video frame images, and their outputs involve the number or presence status of vehicles; therefore, their task structures are also quite similar.

[0094] Based on preset weights (e.g., semantic similarity 0.3, data distribution similarity 0.4, task structure similarity 0.3), the comprehensive correlation score between the two scenarios is calculated.

[0095] Step S6) Select multiple reference scenario models 12 with a correlation degree higher than the preset reference threshold as teaching models.

[0096] For example, target application scenario 21 is "urban traffic flow monitoring". The correlation scores with reference scenario 13 are as follows: Reference scenario 13A "parking lot space detection", correlation score is 0.5. Reference scenario 13B "highway entrance / exit vehicle recognition", correlation score is 0.8. Reference scenario 13C "street pedestrian counting", correlation score is 0.4. Reference scenario 13D "road construction area monitoring", correlation score is 0.7.

[0097] When the preset reference threshold is set to 0.6, the correlation between reference scene 13B and reference scene 13D exceeds the threshold and is selected as the teaching model.

[0098] Step S7) Teach the standard model 22 using multiple teaching models, and after teaching is completed, migrate the standard model 22 to the target device 24.

[0099] The steps of teaching the standard model 22 using multiple of the aforementioned teaching models include:

[0100] Receive sample data of target application scenario 21, construct a unified input interface, and input the sample data of target application scenario 21 into each professor model and the standard model 22;

[0101] The output of each professor model on the sample data is obtained respectively. The output includes the predicted probability distribution of the last layer, the feature representation of the intermediate layer, and the confidence score.

[0102] The outputs of multiple professor models are weighted and fused to obtain a feature knowledge base. The weights of the weighted fusion are obtained based on the correlation between the reference scenario 13 corresponding to the professor model and the target application scenario 21.

[0103] Calculate the loss function, which is obtained based on the output distribution difference loss, intermediate feature alignment loss, and confidence loss;

[0104] The weights of the weighted fusion are dynamically adjusted based on the performance of each professor's model on the current sample data;

[0105] The weights of the standard model 22 are updated using the backpropagation algorithm, so that it gradually approximates the feature knowledge base obtained by weighted fusion of multiple professor models.

[0106] Ensure that all models (professor models and standard model 22) receive input data in a consistent format to facilitate fair comparison and knowledge transfer. Collect real-world sample data (such as images, text, sensor data, etc.) for the target application scenario 21. Construct a unified input interface to preprocess the data (such as normalization, resizing, modality conversion, etc.) to meet the input requirements of all models. Input the processed data into each professor model and standard model 22 simultaneously. For example, the target scenario is "urban traffic flow monitoring," with input being 1080p video frames. Professor model A (from "highway vehicle recognition") accepts 720p input, and professor model B (from "street pedestrian counting") accepts RGB + depth maps. The unified interface needs to downsample the raw data to 720p and use monocular estimation for completion when no depth map is available to ensure all models can run.

[0107] Each teacher model, upon receiving sample data, outputs multi-layered knowledge information, primarily including a predicted probability distribution, intermediate layer feature representations, and a confidence score. The predicted probability distribution is the class probability output by the last softmax layer. The intermediate layer feature representation refers to the activation values ​​of certain key layers (such as the last convolutional layer), reflecting the teacher model's ability to extract abstract features. The confidence score represents the model's reliability in predicting the current outcome (such as the maximum probability value or entropy value), used to measure the teacher's reliability on that sample. For example, teacher model A predicts the probabilities of three types of vehicles in a frame as [0.85, 0.10, 0.05], with an intermediate feature map size of 14×14×256 and a confidence score of 0.85.

[0108] The outputs of multiple professor models are weighted and fused to construct a feature library. Higher correlation results in higher weights, and their outputs have a stronger influence on the standard model. For example, if professor model A has a correlation of 0.8, its weight is set to 0.6. If professor model B has a correlation of 0.7, its weight is set to 0.4. The fused soft label = 0.6 × pA + 0.4 × pB, where pA represents the soft label value of professor model A and pB represents the soft label value of professor model B.

[0109] The loss function consists of three parts, guiding the standard model 22 to approximate the knowledge of the teacher group from multiple dimensions.

[0110] Ltotal=α×Ldistill+β×Lfeature+γ×Lconfidence.

[0111] Ltotal represents the total loss, Ldistill represents the output distribution difference loss, Lfeature represents the intermediate feature alignment loss, and Lconfidence represents the confidence loss. α, β, and γ are coefficients that can be statically set or dynamically adjusted. Some professor models may perform poorly on specific samples (e.g., misclassification, low confidence), in which case their influence should be reduced.

[0112] Backpropagation is performed on the standard model 22 using the total loss Ltotal. Only the parameters of the student models are updated, while all teacher models remain frozen (not updated). After multiple iterations, the standard model 22 gradually incorporates the knowledge from multiple teachers, forming a lightweight, robust, and high-performance model. This model is then transferred to the target device 24 after the teaching is completed.

[0113] On the other hand, this specification provides an adaptive model transfer system based on reverse multi-professor distillation; please refer to the appendix. Figure 4 ,include:

[0114] Reference device 11 module 100 is used to obtain parameters of multiple target devices 24 that may be migrated, and the reference device 11 with the lowest computing power is obtained based on the parameters of the multiple target devices 24;

[0115] Module 200 with reference scenario 13 acquires multiple reference scenarios 13 and corresponding sample data for each reference scenario 13, assigns a reference device 11 to each reference scenario 13, and establishes a model on the reference device 11 whose scale matches the parameters of the reference device 11.

[0116] The reference model module 300 uses the sample data corresponding to each reference scenario 13 to train the corresponding model on the reference device 11. The trained model is denoted as the reference scenario model 12.

[0117] Standard model 22 module 400 establishes a student model based on the parameters of reference device 11, uses a pre-accessed large model as a teacher model to guide the training of the student model, and the trained student model serves as standard model 22.

[0118] The association module 500 obtains the selected target application scenario 21 and obtains the association degree between the target application scenario 21 and each of the reference scenarios 13;

[0119] The selection module 600 selects multiple reference scenario models 12 whose correlation degree is higher than a preset reference threshold as teaching models;

[0120] The migration module 700 uses multiple teaching models to teach the standard model 22, and after the teaching is completed, migrates the standard model 22 to the target device 24.

[0121] Please see Figure 5 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this specification.

[0122] like Figure 5 As shown, the electronic device 1100 may include: at least one processor 1101, at least one network interface 1104, a user interface 1103, a memory 1105, and at least one communication bus 1102. The communication bus 1102 can be used to connect and communicate with the various components mentioned above. The user interface 1103 may include buttons, and optionally may include standard wired or wireless interfaces. The network interface 1104 may include, but is not limited to, a Bluetooth module, an NFC module, or a Wi-Fi module. The processor 1101 may include one or more processing cores. The processor 1101 connects to various parts within the electronic device 1100 using various interfaces and lines, and performs various functions of the routing device and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1105, and by calling data stored in the memory 1105. Optionally, the processor 1101 may be implemented using at least one hardware form of DSP, FPGA, or PLA. The processor 1101 may integrate one or more combinations of CPU, GPU, and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content that the display screen needs to show; and the modem is used for wireless communication.

[0123] It is understandable that the aforementioned modem may not be integrated into the processor 1101, but may be implemented using a separate chip.

[0124] The memory 1105 may include RAM or ROM. Optionally, the memory 1105 may include a non-transitory computer-readable medium. The memory 1105 may be used to store instructions, programs, code, code sets, or instruction sets. The memory 1105 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1105 may also be at least one storage device located remotely from the aforementioned processor 1101. As a computer storage medium, the memory 1105 may include an operating system, a network communication module, a user interface module, and application programs. The processor 1101 may be used to call the application programs stored in the memory 1105 and execute the methods in the above-described embodiments.

[0125] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform multiple steps as described in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the computer-readable storage medium.

[0126] This specification also provides a computer program product, including a computer program that, when executed by a processor, implements the multiple steps described in the above embodiments.

[0127] Where there is no conflict, the technical features in this embodiment and implementation scheme can be combined arbitrarily.

[0128] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes multiple computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center integrating multiple available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state drives (SSDs)).

[0129] When implemented through hardware or firmware, the aforementioned method flow is programmed into the hardware circuit to obtain the corresponding hardware circuit structure and achieve the corresponding function. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit, whose logic function is determined by the user programming the device. Designers can program a digital system onto a PLD themselves, eliminating the need for chip manufacturers to design and fabricate dedicated integrated circuit chips. Furthermore, nowadays, instead of manually fabricating integrated circuit chips, this programming is mostly implemented using "logic compiler" software, similar to the software compiler used in program development. The original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There is not just one HDL, but many. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of the aforementioned hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logic method flow can be easily obtained.

[0130] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims of this specification.

Claims

1. An adaptive model transfer method based on reverse multi-professor distillation, characterized in that, Including the following steps: Obtain parameters of multiple target devices that may be migrated, and obtain a reference device with the lowest computing power based on the parameters of the multiple target devices; Acquire multiple reference scenarios and corresponding sample data for each reference scenario, assign a reference device to each reference scenario, and build a model on the reference device whose scale matches the parameters of the reference device; Each reference scenario's sample data is used to train the corresponding model on the reference device, and the trained model is denoted as the reference scenario model. Based on the parameters of the reference device, a student model is established, and a pre-accessed large model is used as a teacher model to guide the training of the student model. The trained student model is then used as a standard model. Obtain the selected target application scenario and obtain the correlation between the target application scenario and each of the reference scenarios; Multiple reference scenario models with a correlation degree higher than a preset reference threshold are selected as teaching models; The standard model is taught using multiple of the aforementioned teaching models, and after the teaching is completed, the standard model is transferred to the target device.

2. The adaptive model transfer method based on reverse multi-professor distillation according to claim 1, characterized in that, The steps of obtaining parameters of multiple potential target devices and determining the reference device with the lowest computing power based on the parameters of the multiple target devices include: Collect hardware parameter information of the target device, including processor type, number of cores, memory capacity, available storage space, floating-point operation capability, maximum power consumption limit, and supported inference framework; The performance of multiple target devices is normalized, and hardware parameters of different dimensions are unified into performance indicators to obtain the comprehensive computing power evaluation value of the target devices. Based on the comprehensive computing power evaluation value of all target devices, the target device with the lowest comprehensive computing power evaluation value is selected as the reference device.

3. The adaptive model transfer method based on reverse multi-professor distillation according to claim 1 or 2, characterized in that, The steps of establishing a model on the reference device whose scale matches the parameters of the reference device include: An initial model of a preset initial scale is established and run on the reference device, and the running time is recorded. The initial model has a preset number of layers, number of neurons, and weights. When the running time is less than a preset lower reference time, the number of neurons is increased; when the running time is greater than a preset upper reference time, the number of neurons is decreased, until the running time is between the lower reference time and the upper reference time.

4. The adaptive model transfer method based on reverse multi-professor distillation according to claim 1 or 2, characterized in that, The steps for obtaining the correlation between the target application scenario and each of the reference scenarios include: Extract the feature description information of the target application scenario, denoted as the target feature description information. The feature description information includes the input data modality, output index, data distribution characteristics, target device type, and application scenario description. Extract the feature description information for each reference scene, and denote it as the reference feature description information; The target feature description information is compared with the reference feature description information to obtain the correlation between the target application scenario and each reference scenario.

5. The adaptive model transfer method based on reverse multi-professor distillation according to claim 4, characterized in that, The steps to compare the target feature description information with the reference feature description information to obtain the correlation between the target application scenario and each reference scenario include: Semantic similarity is calculated by identifying the semantics of the application scenario descriptions in the target feature description information and the reference feature description information, respectively. Based on the input data modalities and data distribution characteristics of the target feature description information and the reference feature description information, the data distribution similarity is obtained; Based on the input data modalities, output indicators, and data distribution characteristics of the target feature description information and the reference feature description information, the task structure similarity is obtained. The correlation between the target application scenario and each reference scenario is obtained by weighting the semantic similarity, data distribution similarity, and task structure similarity.

6. The adaptive model transfer method based on reverse multi-professor distillation according to claim 1 or 2, characterized in that, The steps for teaching the standard model using multiple of the aforementioned teaching models include: Receive sample data of the target application scenario, construct a unified input interface, and input the sample data of the target application scenario into each professor model and the standard model; The output of each professor model on the sample data is obtained respectively. The output includes the predicted probability distribution of the last layer, the feature representation of the intermediate layer, and the confidence score. The outputs of multiple professor models are weighted and fused to obtain a feature knowledge base. The weights of the weighted fusion are obtained based on the correlation between the reference scenario corresponding to the professor model and the target application scenario. Calculate the loss function, which is obtained based on the output distribution difference loss, intermediate feature alignment loss, and confidence loss; The weights of the weighted fusion are dynamically adjusted based on the performance of each professor's model on the current sample data; The weights of the standard model are updated using the backpropagation algorithm, gradually approximating the feature knowledge base obtained by weighted fusion of multiple professor models.

7. An adaptive model transfer system based on reverse multi-professor distillation, characterized in that, include: The reference device module obtains parameters of multiple target devices that may be migrated, and obtains the reference device with the lowest computing power based on the parameters of the multiple target devices; The reference scenario module acquires multiple reference scenarios and corresponding sample data, assigns a reference device to each reference scenario, and establishes a model on the reference device whose scale matches the parameters of the reference device. The reference model module uses the sample data corresponding to each reference scenario to train the model on the corresponding reference device. The trained model is denoted as the reference scenario model. The standard model module establishes a student model based on the parameters of the reference device, uses a pre-accessed large model as a teacher model to guide the training of the student model, and the trained student model serves as the standard model. The association module obtains the selected target application scenario and the degree of association between the target application scenario and each of the reference scenarios; The selection module selects multiple reference scenario models whose correlation degree is higher than a preset reference threshold as teaching models; The migration module teaches the standard model using multiple teaching models, and after the teaching is completed, migrates the standard model to the target device.

8. An electronic device, characterized in that, Including the processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images