Knowledge distillation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a knowledge distillation method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, in particular to multi-modal artificial intelligence and data processing. The method comprises the steps that a candidate sample data set is screened out from an open domain data set based on a sample data set associated with a target industry and a feature coding layer of a teacher model after parameter fine adjustment, and the data size of the sample data set associated with the target industry is smaller than that of the open domain data set; on the basis of a student model and the teacher model after parameter fine adjustment, aiming at the distillation loss of each candidate sample data in the candidate sample data set, determining the learning value degree of each candidate sample data, and determining distillation training sample data on the basis of the learning value degree; and training a student model based on the distillation training sample data. According to the scheme, the reasoning calculation overhead in the data screening process can be reduced while the high-quality training sample set is rapidly constructed, and the knowledge distillation effect can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the field of multimodal artificial intelligence and data processing technology, and particularly to a knowledge distillation method and apparatus, electronic device, storage medium and program product, which can be applied to scenarios related to industries such as medical care, transportation, finance and law. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, significant progress has been made in open-domain tasks through large-scale model pre-training and multimodal learning. To save computational resources for model deployment, model compression techniques have been proposed, typically achieved through knowledge distillation, quantization, and pruning. Knowledge distillation, a model training technique, aims to distill knowledge from a teacher model with many parameters into a student model with fewer parameters. In practice, the student model is deployed, thus reducing the required computational resources. However, model training usually relies on manual selection of large amounts of sample data from massive open-domain datasets and precise annotation, which suffers from long construction cycles, high manual costs, and difficulty in large-scale scaling. Summary of the Invention

[0003] This disclosure provides a knowledge distillation method, a knowledge distillation apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] In a first aspect, embodiments of this disclosure propose a knowledge distillation method, comprising: selecting candidate sample datasets from an open-domain dataset based on a sample dataset associated with a target industry and a feature encoding layer of a teacher model with fine-tuned parameters, wherein the data volume of the sample dataset associated with the target industry is smaller than the data volume of the open-domain dataset; determining the learning value of each candidate sample data based on the distillation loss of each candidate sample data in the candidate sample dataset based on the student model and the teacher model with fine-tuned parameters, and determining distillation training sample data based on the magnitude of each learning value; and training a student model based on the distillation training sample data.

[0005] Secondly, embodiments of this disclosure propose a knowledge distillation apparatus, comprising: a first screening module, a second screening module, and a knowledge distillation module. The first screening module is configured to select candidate sample datasets from an open-domain dataset based on a sample dataset associated with a target industry and a feature encoding layer of a teacher model with fine-tuned parameters, wherein the data volume of the sample dataset associated with the target industry is less than the data volume of the open-domain dataset. The second screening module is configured to determine the learning value of each candidate sample data based on the distillation loss of each candidate sample data in the candidate sample dataset, using a student model and a teacher model with fine-tuned parameters, and to determine distillation training sample data based on the magnitude of each learning value. The knowledge distillation module is configured to train a student model based on the distillation training sample data.

[0006] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to implement the knowledge distillation method described in any of the above implementations.

[0007] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the knowledge distillation method described in any of the above implementations when executed.

[0008] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, can implement the knowledge distillation method as described in any of the above implementations.

[0009] According to the knowledge distillation scheme provided in this disclosure, a small amount of sample data related to the target industry is used as prior information. Combined with the feature encoding layer of the teacher model after fine-tuning parameters, the sample data of the open domain dataset is coarsely screened to obtain a candidate sample dataset. Compared with the large-scale manual data selection and annotation of sample data construction methods, this method can reduce the data distribution difference between the data in the open domain dataset and the sample data related to the target industry, while reducing the acquisition cost of industry-related data and achieving low-cost and high-efficiency selection of industry-related data. Furthermore, based on the fine-tuned teacher model and the student model to be trained, a secondary screening of the candidate sample dataset can be performed to obtain distilled training sample data for training the student model. Specifically, for each candidate sample data in the candidate sample dataset, the prediction difference between the student model and the fine-tuned teacher model can be determined, i.e., the distillation loss. Then, based on the distillation loss corresponding to each candidate sample data, its learning value can be determined. The distilled training samples are determined according to the magnitude of the learning value corresponding to each candidate sample data. In this way, a high-quality training sample set can be quickly constructed while reducing the inference computation overhead in the data screening process, thereby improving the efficiency of data screening and knowledge distillation, and helping to improve the knowledge distillation effect and the generalization ability of the trained model.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart of a knowledge distillation method provided in this disclosure embodiment; Figure 3 A flowchart of another knowledge distillation method provided in this disclosure embodiment; Figure 4 A schematic diagram of the framework flow of a knowledge distillation method based on an application scenario provided in this embodiment of the disclosure; Figure 5 To and Figure 4 A schematic diagram of the overall process of the corresponding knowledge distillation method; Figure 6 This is a schematic diagram illustrating sample data filtering based on an application scenario, provided in an embodiment of this disclosure. Figure 7A structural block diagram of a knowledge distillation apparatus provided in this disclosure embodiment; Figure 8 This is a schematic diagram of the structure of an electronic device suitable for performing a knowledge distillation method, provided as an embodiment of the present disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding; these should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0013] It should be noted that the collection, acquisition, storage, processing, transmission, provision, disclosure, and application of user personal information (such as medical imaging information, financial report information, contract information, etc.) involved in the technical solution disclosed herein are all carried out with the user's knowledge and explicit authorization, comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0014] With the development of artificial intelligence (AI) technology, significant progress has been made in open-domain tasks in the pre-training of large models (such as large-scale vision-language models) and multimodal learning. The models involved are usually pre-trained based on large-scale open-domain data and have strong general perception and reasoning capabilities. Examples include Large Language and Vision Assistant (LlaVA), Bootstrapping Language-Image Pre-training 2 (BLIP-2), QwenVision Language (QwenVL), InternationalVision Language (InternVL), and other empowered vision-language models.

[0015] Currently, model compression techniques have been proposed to save computational resources for model deployment. These compressions typically involve knowledge distillation, quantization, and pruning. Knowledge distillation, a model training technique, aims to distill knowledge from a teacher model with many parameters into a student model with fewer parameters. In practice, deploying the student model reduces the required computational resources. However, when selecting data for large-scale model pre-training and multimodal learning tasks in relevant industries (such as transportation, healthcare, and remote sensing), manual screening and construction of industry-specific datasets are often necessary. For example, in many scenarios, task-related sample data is manually selected from online or industry databases and precisely labeled. While this ensures data quality and industry relevance, it suffers from long construction cycles, high labor costs, and difficulty in large-scale scaling.

[0016] Furthermore, in the current implicit knowledge distillation process, pre-trained multimodal large models can be directly used as teacher models to infer the entire Web dataset. While this can automate sampling to some extent, the large number of parameters in the teacher model necessitates full inference in each sampling iteration, resulting in extremely high computational and time costs. A single full inference round may involve hundreds of thousands or even millions of samples, and the computational cost and time consumption of GPUs are quite high, which also limits the implementation of rapid iteration. Moreover, existing automated screening methods usually rely on a single metric for sampling, leading to sample selection results that are often biased towards only one dimension. This can easily result in selecting sample data with limited improvement to the model, such as selecting samples that are "industry-relevant but already fully mastered by the student model," resulting in limited training benefits. On the other hand, a single loss metric may select samples that are "difficult for the student model but irrelevant to the industry," leading to insufficient domain or industry fit.

[0017] As can be seen from the above, the data selection and knowledge distillation schemes mentioned above face challenges such as the difficulty in acquiring high-quality data from relevant industries and the high cost of constructing such data. Full-scale inference of the teacher model incurs significant computational and time overhead, thus limiting the iteration speed. Furthermore, the sampling mechanism relies on overly simplistic metrics, which have limited impact on model improvement. Therefore, at least one solution is needed that can ensure the quality of sample data used for knowledge distillation, while reducing computational and time overhead and improving efficiency.

[0018] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the knowledge distillation methods and apparatus, electronic devices, storage media, and program products of this disclosure can be applied.

[0019] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0020] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include data processing applications, knowledge distillation applications, model training applications, and instant messaging applications.

[0021] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0022] Server 105 can provide various services through its built-in applications. Taking knowledge distillation applications as an example, when running such applications, server 105 can achieve the following effects: using a small amount of sample data related to the target industry as prior information, combined with the feature encoding layer of the teacher model after fine-tuning parameters, it can perform a rough screening of sample data in the open domain dataset to obtain a candidate sample dataset. Compared with the large-scale manual data selection and annotation of sample data construction methods, it can reduce the data distribution difference between the data in the open domain dataset and the sample data related to the target industry, while reducing the cost of acquiring industry-related data, and achieving low-cost and high-efficiency selection of industry-related data. Furthermore, based on the fine-tuned teacher model and the student model to be trained, a secondary screening of the candidate sample dataset can be performed to obtain distilled training sample data for training the student model. Specifically, for each candidate sample data in the candidate sample dataset, the prediction difference between the student model and the fine-tuned teacher model can be determined, i.e., the distillation loss. Then, based on the distillation loss corresponding to each candidate sample data, its learning value can be determined. The distilled training samples are determined according to the magnitude of the learning value corresponding to each candidate sample data. In this way, a high-quality training sample set can be quickly constructed while reducing the inference computation overhead in the data screening process, thereby improving the efficiency of data screening and knowledge distillation, and helping to improve the knowledge distillation effect and the generalization ability of the trained model.

[0023] It should be noted that sample data and open domain data related to the target industry can be obtained from terminal devices 101, 102, and 103 via network 104, or they can be pre-stored locally on server 105 through various means. Therefore, when server 105 detects that this data is already stored locally, it can choose to retrieve this data directly from the local storage. In this case, the exemplary system architecture 100 may not include terminal devices 101, 102, and 103 and network 104.

[0024] Since knowledge distillation requires significant computing resources and power, the knowledge distillation methods provided in the subsequent embodiments of this disclosure are generally executed by a server 105 with strong computing power and abundant computing resources. Correspondingly, the knowledge distillation device is also generally located within the server 105. However, it should also be noted that when terminal devices 101, 102, and 103 also possess sufficient computing power and resources, they can also perform the aforementioned calculations performed by the server 105 through the knowledge distillation applications installed on them, thereby outputting the same results as the server 105. Especially when multiple terminal devices with different computing capabilities exist simultaneously, but the knowledge distillation application determines that the terminal device it is on has strong computing power and abundant remaining computing resources, it can allow the terminal device to perform the aforementioned calculations, thereby appropriately reducing the computing pressure on the server 105. Accordingly, the knowledge distillation device can also be located within the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also exclude the server 105 and the network 104.

[0025] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0026] Please refer to Figure 2 , Figure 2 A flowchart of a knowledge distillation method provided in this disclosure embodiment is shown. The knowledge distillation method is executed by an execution entity such as a server, and the method flow 200 may include the following steps: Step 201: Based on the sample dataset associated with the target industry and the feature encoding layer of the teacher model with fine-tuned parameters, select candidate sample datasets from the open domain dataset.

[0027] This step aims to utilize the aforementioned entity performing the knowledge distillation method (e.g., Figure 1 The server 103 shown first coarsely filters out candidate sample datasets from the open-domain dataset based on a sample dataset associated with the target industry and the feature encoding layer of the teacher model after parameter fine-tuning, serving as the basis for subsequent fine-tuning of samples. The sample data associated with the target industry can be a small amount of high-quality, labeled data, with a scale of thousands or less. The open-domain dataset can also be called a general dataset, and the massive amount of Internet (Web) data in this dataset can include data associated with one or more industries. In some optional implementations of this disclosure, the amount of data in the sample dataset associated with the target industry is smaller (or much smaller) than the amount of data in the open-domain dataset.

[0028] In some optional implementations of the embodiments of this disclosure, the sample data associated with the target industry can be used to fine-tune the parameters of the teacher model so that the embedding space (EembeddingSpace refers to the high-dimensional space where the vectors are located after representing the corresponding objects (such as words, images, sentences, audio segments, etc.)) of the teacher model after parameter fine-tuning is aligned with the relevant data of the target industry. In this way, candidate sample datasets aligned with the sample datasets associated with the target industry can be selected from the open-domain dataset based on the teacher model with fine-tuned parameters. Furthermore, the candidate sample datasets can be selected through the feature encoding layer of the teacher model with fine-tuned parameters, which can also improve the screening efficiency in the data coarse-screening stage. For example, a lightweight, pre-trained multimodal model can be selected as the teacher model (or expert model), such as MobileVLM v2, Mixture of Experts-Large Language and Vision Assistant (MoE-LlaVA), etc. In some optional implementations of the embodiments of this disclosure, the parameters of one or more of the feature encoding layer and multimodal fusion part of the teacher model can be fine-tuned based on sample data associated with the target industry to obtain a teacher model with fine-tuned parameters. The feature encoding layer of the teacher model with fine-tuned parameters can be a visual feature encoding layer, a text feature encoding layer, a combination of a visual feature encoding layer and a text feature encoding layer, or it can be used in combination with a feature encoding layer that encodes features from other dimensions. The other dimensions described here should be dimensions related to the function of the teacher model, such as a semantic feature encoding layer, a hierarchical distribution feature encoding layer, etc., and should not include dimensions that are not related to the function of the teacher model.

[0029] In some optional implementations of the embodiments of this disclosure, the sample data in the sample dataset associated with the target industry includes, but is not limited to, multimodal data associated with the target industry, to ensure the diversity of the sample data, thereby ensuring a more comprehensive and accurate model and improving the robustness and generalization ability of the model. For example, the multimodal data associated with the target industry includes, but is not limited to, image data, text data, and audio data.

[0030] In some optional implementations of the embodiments of this disclosure, the sample datasets associated with the target industry mentioned above include, but are not limited to, at least one of the following: sample datasets for generating medical diagnostic reports, sample datasets for retrieving medical cases, sample datasets for recognizing autonomous driving scenarios, sample datasets for detecting traffic monitoring anomalies, sample datasets for answering traffic safety questions, sample datasets for financial statement analysis, sample datasets for risk control analysis, sample datasets for intelligent investment research questions, sample datasets for contract review, sample datasets for retrieving judicial cases, and sample datasets for answering legal and regulatory questions.

[0031] In this embodiment, for example, the sample data in the sample datasets for generating medical diagnostic reports and for retrieving medical cases may include, but is not limited to, at least one of high-quality (e.g., high-resolution) image data (such as computed tomography (CT), magnetic resonance imaging (MRI), pathological slides, etc.) and report text data. The sample data in the sample datasets for autonomous driving scene recognition, traffic monitoring anomaly detection, and traffic safety knowledge Q&A may include, but is not limited to, at least one of road scene images, accident cases, traffic signs, and in-vehicle videos. The sample data in the sample datasets for financial statement analysis, risk control analysis, and intelligent investment research Q&A may include, but is not limited to, financial research reports, financial announcements, and images. The sample data in the sample datasets for contract review, judicial case retrieval, and legal and regulatory Q&A may include, but is not limited to, at least one of judicial cases, contract data, and legal and regulatory interpretation data.

[0032] Step 202: Based on the student model and the teacher model with fine-tuned parameters, determine the learning value of each candidate sample data for the distillation loss of each candidate sample data in the candidate sample dataset, and determine the distillation training sample data based on the magnitude of each learning value.

[0033] Step 203: Train the student model based on the distillation training sample data.

[0034] Based on step 201, the aforementioned executing entity of the knowledge distillation method (e.g., Figure 1The server 103 shown first determines the learning value of each candidate sample data based on the teacher model with the fine-tuned parameters and the initial student model to be trained, according to the distillation loss of each candidate sample data in the candidate sample dataset. Then, it can determine the distillation training sample data based on the magnitude of all learning value corresponding to all candidate sample data in the candidate sample dataset, so as to realize the fine screening of sample data based on the candidate sample dataset, and obtain the sample data used to train the student model, thereby realizing the training of the student model based on the sample data.

[0035] In this embodiment, the distillation loss of each candidate sample data can be used to characterize the difference between the outputs obtained by the student model and the teacher model after fine-tuning the parameters when predicting each candidate sample data. The learning value of each candidate sample data can be used to characterize the likelihood or probability that the candidate sample data can be selected for training the student model. In this way, it can be ensured that the distilled training sample data selected based on the learning value of each sample data in the candidate sample dataset is more conducive to training the student model that is related to the target industry. That is, high-value samples that can effectively make up for the shortcomings of the student model are selected, so that the student model trained based on the distilled training sample data is more suitable for the target industry.

[0036] The knowledge distillation method provided in this disclosure uses a small amount of sample data related to the target industry as prior information. Combined with the feature encoding layer of the teacher model after fine-tuning parameters, it performs a coarse screening of sample data in the open domain dataset to obtain a candidate sample dataset. Compared with the large-scale manual data selection and annotation of sample data construction methods, it can reduce the data distribution difference between the data in the open domain dataset and the sample data related to the target industry, while reducing the acquisition cost of industry-related data and achieving low-cost and high-efficiency selection of industry-related data. Furthermore, based on the fine-tuned teacher model and the student model to be trained, a secondary screening of the candidate sample dataset can be performed to obtain distilled training sample data for training the student model. Specifically, for each candidate sample data in the candidate sample dataset, the prediction difference between the student model and the fine-tuned teacher model can be determined, i.e., the distillation loss. Then, based on the distillation loss corresponding to each candidate sample data, its learning value can be determined. The distilled training samples are determined according to the magnitude of the learning value corresponding to each candidate sample data. In this way, a high-quality training sample set can be quickly constructed while reducing the inference computation overhead in the data screening process, thereby improving the efficiency of data screening and knowledge distillation, and helping to improve the knowledge distillation effect and the generalization ability of the trained model.

[0037] Please refer to Figure 3 , Figure 3A flowchart of another method for analyzing tennis ball sports videos provided in this disclosure embodiment, wherein process 300 includes the following steps: Step 301: Based on the sample dataset associated with the target industry and the feature encoding layer of the teacher model after fine-tuning the parameters, select candidate sample datasets from the open domain dataset, wherein the data volume of the sample dataset associated with the target industry is smaller than the data volume of the open domain dataset; Step 302: Based on the student model and the teacher model with fine-tuned parameters, determine the learning value of each candidate sample data by applying the distillation loss to the candidate sample dataset, and determine the distillation training sample data based on the magnitude of each learning value. Steps 301-302 above and as follows Figure 2 Steps 201-202 shown are the same; for the same parts, please refer to the corresponding parts of the previous embodiment, and will not be repeated here. Optionally, in this embodiment, the above... Figure 2 Step 203 in the corresponding embodiment can be implemented through the following steps 303-305, which may specifically include: Step 303: Based on the distillation training sample data, determine the first output feature of the student model in the target layer and the second output feature of the teacher model in the target layer after fine-tuning the parameters, wherein the target layer is determined based on the process of fine-tuning the parameters of the teacher model.

[0038] Step 304: Based on the first output feature and the second output feature, determine the feature alignment loss of the student model and the teacher model after fine-tuning the parameters for the target layer.

[0039] Step 305: Train the student model based on feature alignment loss.

[0040] In this embodiment, after distilling training sample data to select valuable training data using a two-stage sample selection mechanism, the first output feature of the student model at its target layer and the second output feature of the teacher model at its target layer after parameter tuning can be determined for the same distilled training sample data. Based on this, the feature alignment loss for the student model and the teacher model after parameter tuning at the target layer can be determined. This allows for explicit knowledge distillation at the target layer during model training based on this feature alignment loss, effectively addressing the problem of insufficient industry data and reducing the cost of training large models. The number of distilled training sample data can be one or more.

[0041] In some optional implementations of the embodiments of this disclosure, the aforementioned target layer may refer to the layer where the parameters change most drastically during the fine-tuning of the teacher model's parameters, the core or hotspot areas where the model learns key information to adapt to new tasks, and these areas are then defined as the key layer for knowledge distillation, i.e., the target layer. Optionally, the target layer includes, but is not limited to, a feature encoding layer, wherein the feature encoding layer may specifically be a visual encoding layer, which helps to fully capture the visual features unique to the target industry; or the feature encoding layer may specifically be a text encoding layer, which helps to fully capture the semantic features unique to the target industry; or the feature encoding layer may specifically be a combination of a visual encoding layer and a text encoding layer, which helps to fully capture the semantic and visual features unique to the target industry. Of course, it can also be a combination of any of the above encoding layers with encoding layers of other dimensions.

[0042] In some optional implementations of the embodiments of this disclosure, the target layer may further include the feature output layer of the model. In this case, when determining the total training loss, in addition to considering the feature alignment loss and the supervision loss of the student model, the output alignment loss of the target layer may also be determined based on the distilled training sample data, combined with the student model and the teacher model after fine-tuning the parameters.

[0043] In some optional implementations of the embodiments of this disclosure, step 305 above can be implemented as follows: determining the total training loss based on the weighted sum of the feature alignment loss and the supervised loss corresponding to the student model; and training the student model based on the total training loss.

[0044] In this embodiment, when determining the total training loss for training the student model, the total training loss can be determined by weighted summation of the feature alignment loss of the student model and the teacher model after fine-tuning the parameters for the target layer, as well as the supervision loss corresponding to the student model. This total training loss can then be used to train the student model until the conditions for training completion are met, thereby improving the efficiency and reliability of model training.

[0045] In some optional implementations of the embodiments of this disclosure, the conditions for the completion of the training of the student model may include, but are not limited to, the validation set loss no longer decreasing.

[0046] In some optional implementations of the embodiments of this disclosure, the above Figure 2 Step 201 in the corresponding embodiment or Figure 3Step 301 in the corresponding embodiment can be implemented as follows: The feature encoding layer of the teacher model, after fine-tuning the parameters of each sample data in the sample dataset associated with the target industry, determines the first embedding vector corresponding to each sample data; the feature encoding layer of the teacher model, after fine-tuning the parameters of each data in the open domain dataset, determines the second embedding vector corresponding to each data; based on each second embedding vector and each first embedding vector, the similarity between each data in the open domain dataset and each sample data in the sample dataset associated with the target industry is determined; based on the magnitude of each similarity, candidate sample datasets are selected from the open domain dataset.

[0047] In this embodiment, a finely tuned teacher model can be used to measure industry similarity or relevance of data in massive internet data, i.e., open-domain datasets. This allows for the priority selection of samples from the open-domain dataset whose distribution is similar to that of sample datasets associated with the target industry. In other words, industry similarity or relevance can be used as a driving force to perform a coarse screening of the open-domain dataset, helping to mitigate the impact of domain differences on knowledge distillation. Specifically, sample data from sample datasets associated with the target industry and data from the open-domain dataset can be processed by the finely tuned teacher model to obtain corresponding first and second embedding vectors. Based on these two vectors, the similarity between the corresponding sample data related to the target industry and the data in the open-domain dataset can be determined. This similarity can be used to measure industry similarity or relevance of each data point in the open-domain dataset. Furthermore, based on the magnitude of all obtained similarity scores, a coarse screening of the data in the open-domain dataset can be achieved, resulting in a candidate sample dataset.

[0048] In some optional implementations of the embodiments of this disclosure, the step of selecting candidate sample datasets from the open domain dataset based on the magnitude of each similarity can be implemented as follows: for each data in the open domain dataset, select first sample data whose similarity magnitude meets the first preset condition from the sample dataset associated with the target industry; determine the relevance score corresponding to each data in the open domain dataset based on the similarity corresponding to each first sample data; and select candidate sample datasets from the open domain dataset based on the magnitude of the relevance score corresponding to each data in the open domain dataset.

[0049] In this embodiment, since the open domain dataset contains data from various industries and fields, for a selected or known target industry, the similarity between each data point and the target industry can be determined by calculating the similarity between each data point and the sample data in the sample dataset associated with that target industry. Then, based on the industry similarity, first sample data meeting a first preset condition is selected. That is, each data point in the open domain dataset corresponds to its own first sample data. The number of first sample data can be one or more. Optionally, the first preset condition includes, but is not limited to, the maximum similarity value or the similarity value being greater than a preset similarity threshold. Furthermore, a first preset proportion of sample data meeting the aforementioned condition can be set, without specific limitations. The size of the preset similarity threshold and the first preset proportion can be set according to specific circumstances, without specific limitations. Then, for each data point in the open domain dataset, its relevance score can be determined based on the similarity of all its corresponding first sample data, thereby achieving industry similarity or relevance measurement of each data point in the open domain dataset. Then, based on the magnitude of the relevance score corresponding to each data point in the open domain dataset, candidate sample datasets can be selected from the open domain dataset. In this way, by quickly screening samples that are similar in distribution to the target industry through the first stage of coarse screening, the inference and computational overhead of the data screening process can be significantly reduced.

[0050] As can be seen from the above, in this embodiment, by adopting a two-stage sample selection mechanism, a dual selection mechanism that simultaneously focuses on the relevance of sample data to the target industry and the learnability of the student model is achieved. This ensures that the sample data is strongly correlated with the target industry while prioritizing the selection of samples with the greatest teaching or learning value. This not only significantly reduces the interference of invalid or noisy data but also enables the rapid construction of a high-quality training set under conditions of extremely scarce industry data, thereby quickly training a student model that excels in the target industry (or field).

[0051] In some optional implementations of the embodiments of this disclosure, for each data in the open domain dataset, the average similarity of all the corresponding first sample data can be calculated, and the calculation result can be used as the relevance score of each data in the open domain dataset.

[0052] In some optional implementations of the embodiments of this disclosure, the above Figure 2 Step 202 in the corresponding embodiment or Figure 3Step 302 in the corresponding embodiment can be implemented as follows: the result of weighted summation of the distillation loss and correlation score corresponding to each candidate sample data is determined as the learning value of each candidate sample data; second sample data that meets the second preset condition in terms of learning value is selected from the candidate sample dataset; and distillation training sample data is determined based on the second sample data.

[0053] In this embodiment, based on the relevance scores of each data point in the open-domain dataset and the preliminary selection of candidate sample datasets from the open-domain dataset, it is known that each candidate sample in the candidate sample dataset has its own relevance score. Then, combining the distillation loss corresponding to each candidate sample data—that is, the difference between the predictions of the student model and the teacher model after fine-tuning parameters on each candidate sample data—a weighted summation operation is performed to determine the learning value of each candidate sample in the candidate sample dataset. Further, based on the second sample data selected from the candidate sample dataset whose learning value meets the second preset condition, distillation training sample data for training the student model can be determined. In this way, the knowledge gaps in the student model can be identified, and high-value samples that are both relevant to the target industry and can effectively compensate for the deficiencies of the student model can be selected or finely screened from the preliminary candidate sample dataset, thereby improving the distillation effect and the model's generalization ability.

[0054] Optionally, the aforementioned second preset condition includes, but is not limited to, the maximum value of the learning value or the value of the learning value being greater than the preset learning value threshold. Furthermore, sample data that meets the aforementioned second preset ratio can also be set on this basis, without specific limitations. The size of the preset learning value threshold and the second preset ratio can be set according to specific circumstances, without specific limitations.

[0055] In some optional implementations of the embodiments of this disclosure, the step of determining distillation training sample data based on the second sample data can be implemented as follows: combining the second sample data with sample data in a sample dataset associated with the target industry, and determining the combined sample data as distillation training sample data.

[0056] In this embodiment, in order to further improve the distillation effect and the generalization ability of the model, the second sample data selected from the above candidate sample dataset that meets the second preset condition, and the dataset composed of the aforementioned small amount of high-quality sample data related to the target industry, can be determined as the distillation training sample data.

[0057] In some optional implementations of any of the above embodiments of this disclosure, the knowledge distillation method may further include: first, determining a first loss of the student model for each candidate sample data, and determining a second loss of the teacher model after fine-tuning parameters for each candidate sample data; then determining the difference between the first loss and the second loss; and finally, determining the distillation loss based on the difference and the confidence weight corresponding to the teacher model after fine-tuning parameters.

[0058] In this embodiment, after determining the student model and the teacher model with fine-tuned parameters, when performing distillation loss on each candidate sample data in the candidate sample set, in order to further ensure the effective identification of the knowledge weaknesses of the student model, high-value samples that are both close to the target industry and can effectively compensate for the deficiencies of the student model are selected or finely screened from the candidate sample dataset obtained after coarse screening, thereby improving the distillation effect and the generalization ability of the model. This can be achieved by further considering the confidence weights corresponding to the teacher model with fine-tuned parameters, thereby increasing the probability that high-value sample data is finely screened. In some optional implementations of this disclosure, the first loss and second loss used to determine the above difference include, but are not limited to, the loss of the output layer of each model when inputting each candidate sample data into the corresponding model.

[0059] In some optional implementations of the embodiments of this disclosure, the step of determining the distillation loss based on the difference and the confidence weight corresponding to the teacher model after fine-tuning the parameters can be implemented as follows: in response to determining that the difference is greater than a preset value, the result of multiplying the difference and the confidence weight is determined as the distillation loss; or in response to determining that the difference is less than a preset value, the result of multiplying the preset value and the confidence weight is determined as the distillation loss.

[0060] In this embodiment, when determining the distillation loss of the student model and the teacher model after fine-tuning parameters for each candidate sample data in the candidate sample set, in order to further ensure the effective identification of the knowledge weaknesses of the student model, high-value samples that are both close to the target industry and can effectively compensate for the deficiencies of the student model are selected or finely screened from the candidate sample dataset obtained after coarse screening, thereby improving the distillation effect and the generalization ability of the model. The larger of the above difference and the preset value can be used as the result of the score calculation with the confidence weight corresponding to the teacher model after fine-tuning parameters, and determined as the distillation loss. The value of the preset value can be set according to the specific situation, and is not specifically limited here. For example, the preset value can be set to 0.

[0061] In any of the discovery methods of this disclosure, the parameters that can be fine-tuned by the teacher model may include, but are not limited to, the proportion of sample data coarsely screened from the open domain dataset based on the above-mentioned industry relevance, the proportion of sample data finely screened from the candidate sample dataset based on the degree of learning value, and the weight parameters used to obtain the total training loss by weighted summation, etc., which can be adjusted according to actual needs and are not specifically limited here.

[0062] To deepen understanding, this disclosure also presents a specific implementation scheme based on a concrete application scenario, proposing a two-stage data distillation framework driven by industry alignment and learnability. This framework aims to address the problems of low efficiency, high computational cost, and limited sample value assessment in multimodal industry tasks. Through a two-stage data screening mechanism of coarse and fine screening, this scheme balances industry relevance with the knowledge gain of the student model, significantly reducing computational overhead while ensuring screening quality.

[0063] In this approach, industry data can be prepared in advance, obtaining a small amount of high-quality labeled datasets from industry tasks. (The sample dataset associated with the target industry in the above embodiments) can be in the thousands or fewer in size. This data, as "industry prototype samples," can be used for industry alignment fine-tuning of the expert model (corresponding to the teacher model in the above embodiments).

[0064] Please see Figure 4 and Figure 5 It mainly includes the following processes: Step 401: Expert model construction and industry alignment fine-tuning; Optionally, a lightweight multimodal model can be selected as the expert model (such as MobileVLM v2, MoE-LLaVA), and a small number of parameters can be fine-tuned in the visual encoder and multimodal fusion part of the expert model to align its embedding space distribution with industry data, so as to obtain the expert model after fine-tuning parameters (corresponding to the teacher model after fine-tuning parameters in the above embodiment).

[0065] By using the above labeled dataset The data input to the expert model after fine-tuning the parameters is used to output a visual-text embedding vector, denoted as . (Corresponding to the first embedding vector in the above embodiment).

[0066] Step 402: Coarse screening of sample data based on industry similarity or relevance; For large-scale open-domain Web datasets Using publicly available data, the embedding vectors are calculated using the visual encoder of an expert model with finely tuned parameters (which may be combined with a text encoder if necessary, corresponding to the feature encoding layer in the above embodiments): (Corresponding to the second embedding vector in the above embodiment).

[0067] Calculate the similarity between web data and industry prototype embeddings in the embedding space, such as cosine similarity: ; Where cos( , ) represents the embedding vector and The cosine similarity between them, || represents the modulo operation. For each Web sample i (corresponding to each data in the open domain dataset in the above embodiment), from the above labeled dataset... Find the r industry samples with the highest similarity to it, that is, select the top-r nearest neighbors and store their indices in a set. : ; The average similarity of the r nearest neighbor industry samples is used as the industry relevance score for Web sample i: ; Finally, based on industry relevance scoring Open Domain Web datasets sorted from highest to lowest Of all web samples, those with the highest industry relevance scores were selected. The sample constitutes the candidate set (Corresponding to the candidate sample dataset in the above embodiments), similarity-based screening results are obtained for use in the subsequent fine screening stage.

[0068] Step 403: Fine-tuning of sample data based on learnability; Use the student model to process the candidate set. The sample data is used for inference, and a measure of the predictive difference between the model and the teacher model (i.e., the aforementioned expert model) is calculated (or the industry-task-based distillation loss, which can be used to characterize learnability scores / points), which can be expressed as: ; in, This represents the student model's loss on sample x (referred to as student loss). This represents the expert model's loss on sample x (referred to as expert loss). Optionally, this loss can refer to the loss corresponding to the model's output layer. ,as well as The confidence weights of the teacher model are generally taken as... ,in, H is the predicted distribution (classification / vocabulary distribution) of the teacher model on sample x. H() refers to the normalized entropy (i.e. uncertainty). The flatter the distribution, the greater the entropy. The sharper the distribution (i.e. confidence), the smaller the entropy. It means that the "uncertainty" is treated as "confidence". This increases the contribution of samples that the teacher model is very confident about and decreases the contribution of samples that the teacher model is not sure about. That is, the low entropy of the teacher model's predicted distribution indicates high confidence.

[0069] Then, the industry relevance score and learnability score are weighted and combined to form a merged score: ; in, Indicates the weights, and selects the top-ranked items with the highest combined scores after weighted combination. sample set Used to obtain distillation training data. In this embodiment, the distillation training data includes, in addition to the sample set, In addition to the data in the dataset, the labeled datasets mentioned above can also be included. The data in the middle.

[0070] Step 404: Explicit knowledge distillation; Taking the fine-tuning phase of a teacher model like LLaVA as an example, the parameters of both the visual and language modules are updated to adapt to the downstream task. Therefore, the layers where the parameters change most drastically can be considered the core regions where the model learns key information to adapt to the new task.

[0071] In this embodiment, a method is proposed to identify "hotspot regions (or core regions)" that show significant changes during fine-tuning through a one-time exploratory experiment, and then define these regions as the key layer set for knowledge distillation. (Corresponding to the target layer in the above embodiments), forming a simple, efficient and well-founded fixed strategy.

[0072] Then, using the selected sample set For the aforementioned key layer set In each layer, the feature alignment loss between the student model and the teacher model is calculated. : ; in, and The student model and the teacher model were respectively in the first... The layer's feature output, || represents modulo, and the feature alignment loss. This can be used to obtain the following total training loss: ; This includes distillation losses (which may include distillation losses corresponding to the output layer). The distillation loss corresponding to the feature layer is the feature alignment loss mentioned above. KD represents the output layer, and feat represents the feature layer) and the supervision loss of the student model ( We perform a weighted summation to obtain the final total training loss. ,in, , , Indicates the weight.

[0073] In this embodiment, the data flow can be described as follows: the input data is a small industry sample set + an open-domain Web dataset; during the intermediate processing of the data, including but not limited to: the expert model uses the small industry sample set to fine-tune its parameters, the expert model after fine-tuning its parameters is used to output the corresponding embedding vector based on the industry sample data, and to perform the corresponding embedding vector calculation based on the Web data, thereby realizing the ANN coarse screening, then performing the sample fine screening based on the expert model and the student model, and then using the data obtained from the fine screening to perform explicit knowledge distillation in the key layer, the output data being a distilled sample set that is industry-related and has knowledge gain value; and based on this, the training of the industry-adapted student multimodal model is completed.

[0074] The solution in this disclosure employs a dual screening mechanism of "industry similarity + learnability" to prioritize the most valuable teaching samples while ensuring strong industry relevance. This significantly reduces interference from invalid or noisy data and enables the rapid construction of a high-quality training set even under conditions of extremely scarce industry data, thereby quickly training a student model that excels in the target domain. This solution can significantly improve the training efficiency and effectiveness of large models in specific industry scenarios. Compared to solutions that require more annotations or longer training cycles, this solution can achieve industry-specific model enhancement at a lower cost and in a shorter time. By using a small amount of manually selected industry data to train a lightweight expert model, it replaces large-scale manual data selection and annotation, achieving low-cost and high-efficiency selection of industry-related data. This reduces computational costs and the costs of acquiring and constructing industry data. For example, in the initial screening stage, only visual models and text embeddings are used to calculate similarity, while inference is only run after fine screening. This reduces the number of full inference iterations of the large model and improves the quality of sample selection. Simultaneously, considering industry relevance and learnability, it ensures that the samples are both task-appropriate and beneficial to the student model. Moreover, the solutions provided in this disclosure are highly practical and can be deployed in resource-constrained industry scenarios, supporting rapid iteration of multimodal tasks.

[0075] The solution in this disclosure aims to reduce the data distribution differences between open domains and industry tasks. It utilizes expert models to measure industry similarity across massive amounts of internet data, prioritizing samples with distributions similar to industry data to mitigate the impact of domain differences on distillation performance. Furthermore, to significantly reduce the inference computation overhead of the data screening process, a two-stage screening mechanism is employed. In the coarse screening stage, only the visual encoder of the expert model is used to calculate embedding vectors, and candidate samples with embeddings similar to industry data are quickly screened using Approximate Nearest Neighbor (ANN) retrieval, greatly reducing computational load. High-cost teacher model inference is only invoked in the fine screening stage, significantly reducing the number of full inference iterations and improving screening and distillation efficiency. Additionally, a sampling mechanism that simultaneously considers industry relevance and student model learnability is implemented. In the fine screening stage, based on the difference in predicted distributions between the expert model and the student model on candidate samples, weak knowledge areas in the student model are identified. High-value samples that are both close to the industry task and effectively compensate for the student model's shortcomings are selected, thereby improving distillation performance and model generalization ability, and significantly enhancing the training efficiency and effectiveness of large models in specific industry scenarios.

[0076] The aforementioned two-stage multimodal data screening and distillation framework can be widely applied in fields such as healthcare, transportation, finance, law, e-commerce, and education. Its value lies in: utilizing a small amount of industry data as prior information, screening data based on industry similarity and learnability, and performing knowledge distillation based on "hotspot areas" identified during fine-tuning. This effectively solves the problem of "insufficient industry data" and reduces the "training cost of large models." The interrelationships between various data types can be found in [reference needed]. Figure 6 While internet data has broad coverage, it is also noisy, whereas industry data is relatively scarce and precise; the intersection of the two (such as...) Figure 6 The internet-industry data intersection shown contains some relevant data, but there are still biases. It's necessary to first filter by industry similarity to gradually narrow down the scope, ultimately selecting data that is relatively close to the industry data (such as...). Figure 6 The data distribution after similarity filtering is shown in the figure. Furthermore, performance differences or data distribution differences exist between the teacher-student models in the intersection region. This "learnability" can be used to further filter out samples with greater educational value, minimizing the difference between the data distribution of the internet-industry data intersection and the distribution of the selected industry data.

[0077] After obtaining the initial data through industry similarity screening, it is necessary to further utilize the performance differences between teacher and student models to determine the value of the samples: if both teacher and student models can learn, the training benefits are limited; if the teacher model cannot learn, it may be noise or an extremely difficult sample, which should be removed; only samples that the teacher model can learn but the student model cannot truly reflect "learnability" and are the most valuable for teaching and the most worthy of priority retention in fine screening.

[0078] In the healthcare industry, high-quality imaging data (CT, MRI, pathology slides) and report texts exist, but the scale of real-world industry data is limited due to privacy and compliance issues. Therefore, a small number of industry images and reports can be embedded as industry prototypes, and similar image and text data (such as publicly available medical images and medical educational materials) can be selected from large-scale open-domain Web data. Then, the selected data and industry data can be used together for distillation training to enhance the model's adaptability to medical tasks. For example, this can be applied to assisting in the generation of diagnostic reports and the retrieval of image-similar cases.

[0079] In the transportation industry, data is characterized by a large amount of publicly available traffic-related data online, such as road images, traffic signs, in-vehicle videos, and news reports, with relatively weak data privacy constraints. Therefore, industry-specific embeddings can be used to filter traffic-related images and text from web data (such as road scenes, accident cases, and traffic regulation descriptions) for distillation training, focusing on aligning "scene recognition—rule understanding" capabilities. For example, this can be applied to autonomous driving scene recognition, traffic monitoring anomaly detection, and traffic safety knowledge Q&A.

[0080] In the financial industry, data is characterized by copyright issues for some industry-specific data (research reports, announcements, charts), high timeliness requirements, and a large volume of publicly available information on the web (news, financial websites, forums). Therefore, research report summaries and financial statement texts can be used as industry embeddings to filter financial news and chart data from the web. Furthermore, cross-modal (chart-text) alignment can be emphasized during distillation training to improve the model's understanding of industry reports. This approach can be applied to automatic financial statement parsing, intelligent investment research Q&A, and financial risk control analysis.

[0081] In the legal industry, data is characterized by privacy and security restrictions on judicial cases and contract data, while some cases are publicly available, and there is a wealth of legal interpretations and regulatory explanations on the web. Therefore, a small amount of real-world case data can be used to construct an industry-specific embedding, and samples of regulatory interpretations and contract clause explanations can be selected from the web to expand the dataset. Furthermore, during distillation, emphasis should be placed on language-level alignment to ensure consistency between professional terminology and industry logic. This can be applied to intelligent contract review, case retrieval, and regulatory Q&A.

[0082] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a knowledge distillation apparatus, which is similar to... Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to electronic devices such as servers.

[0083] like Figure 7As shown, the knowledge distillation apparatus 700 of this embodiment may include: a first screening module 701, a second screening module 702, and a knowledge distillation module 703. The first screening module 701 is configured to screen candidate sample datasets from an open-domain dataset based on a sample dataset associated with a target industry and the feature encoding layer of a teacher model with fine-tuned parameters, wherein the data volume of the sample dataset associated with the target industry is smaller than the data volume of the open-domain dataset. The second screening module 702 is configured to determine the learning value of each candidate sample data based on the distillation loss of each candidate sample data in the candidate sample dataset, based on the student model and the teacher model with fine-tuned parameters, and determine the distillation training sample data based on the magnitude of each learning value. The knowledge distillation module 703 is configured to train the student model based on the distillation training sample data.

[0084] In this embodiment of the disclosure, the specific processing of the first screening module 701, the second screening module 702, and the knowledge distillation module 703 in the knowledge distillation apparatus 700, and the resulting technical effects, can be found in reference to [the relevant documentation]. Figure 2 The relevant descriptions of steps 201-203 in the corresponding embodiments will not be repeated here.

[0085] In some optional implementations of the embodiments of this disclosure, the first screening module 701 is further configured to: determine a first embedding vector corresponding to each sample data by passing the feature encoding layer of the teacher model after fine-tuning the parameters of each sample data in the sample dataset associated with the target industry; determine a second embedding vector corresponding to each data by passing the feature encoding layer of the teacher model after fine-tuning the parameters of each data in the open domain dataset; determine the similarity between each data in the open domain dataset and each sample data in the sample dataset associated with the target industry based on each second embedding vector and each first embedding vector; and select candidate sample datasets from the open domain dataset based on the magnitude of each similarity.

[0086] In some optional implementations of the embodiments of this disclosure, the first screening module 701 is further configured to: for each data in the open domain dataset, select first sample data whose similarity satisfies a first preset condition from the sample dataset associated with the target industry; determine the relevance score corresponding to each data in the open domain dataset based on the similarity corresponding to each first sample data; and select candidate sample datasets from the open domain dataset based on the magnitude of the relevance score corresponding to each data in the open domain dataset.

[0087] In some optional implementations of the embodiments of this disclosure, the second screening module 702 is further configured to: determine the learning value of each candidate sample data by weighted summation of the distillation loss and correlation score corresponding to each candidate sample data; screen out second sample data whose learning value satisfies the second preset condition from the candidate sample dataset; and determine distillation training sample data based on the second sample data.

[0088] In some optional implementations of the embodiments of this disclosure, the second screening module 702 is further configured to: combine the second sample data and the sample data in the sample dataset associated with the target industry, and determine the combined sample data as distillation training sample data.

[0089] In some optional implementations of the embodiments of this disclosure, the knowledge distillation apparatus 700 described above may further include a determining module (not shown in the figure), configured to: determine a first loss of the student model for each candidate sample data; determine a second loss of the teacher model after fine-tuning parameters for each candidate sample data; determine the difference between the first loss and the second loss; and determine the distillation loss based on the difference and the confidence weight corresponding to the teacher model after fine-tuning parameters.

[0090] In some optional implementations of the embodiments of this disclosure, the above-mentioned determining module is further configured to: in response to determining that the difference value is greater than a preset value, determine the result of multiplying the difference value and the confidence weight as the distillation loss; or in response to determining that the difference value is less than a preset value, determine the result of multiplying the preset value and the confidence weight as the distillation loss.

[0091] In some optional implementations of the embodiments of this disclosure, the knowledge distillation module 703 is further configured to: determine the first output feature of the student model in the target layer and the second output feature of the teacher model after fine-tuning parameters in the target layer based on the distillation training sample data, wherein the target layer is determined based on the process of fine-tuning parameters of the teacher model; determine the feature alignment loss of the student model and the teacher model after fine-tuning parameters for the target layer based on the first output feature and the second output feature; and train the student model based on the feature alignment loss.

[0092] In some optional implementations of the embodiments of this disclosure, the knowledge distillation module 703 is further configured to: determine the total training loss based on the weighted summation of the feature alignment loss and the supervised loss corresponding to the student model; and train the student model based on the total training loss.

[0093] In some optional implementations of the embodiments of this disclosure, the sample data in the sample dataset associated with the target industry includes multimodal data associated with the target industry.

[0094] In some optional implementations of the embodiments of this disclosure, the sample dataset associated with the target industry mentioned above includes at least one of the following: a sample dataset for generating medical diagnostic reports, a sample dataset for retrieving medical cases, a sample dataset for recognizing autonomous driving scenarios, a sample dataset for detecting traffic monitoring anomalies, a sample dataset for answering traffic safety questions, a sample dataset for financial statement analysis, a sample dataset for risk control analysis, a sample dataset for intelligent investment research questions, a sample dataset for contract review, a sample dataset for retrieving judicial cases, and a sample dataset for answering legal and regulatory questions.

[0095] This embodiment exists as a device embodiment corresponding to the above method embodiment. It uses a small amount of sample data related to the target industry as prior information, and combines it with the feature encoding layer of the teacher model after fine-tuning the parameters to achieve a rough screening of sample data in the open domain dataset to obtain a candidate sample dataset. Compared with the large-scale manual data selection and annotation of sample data construction methods, it can reduce the data distribution difference between the data in the open domain dataset and the sample data related to the target industry, while reducing the cost of acquiring industry-related data, and achieving low-cost and high-efficiency selection of industry-related data. Furthermore, based on the fine-tuned teacher model and the student model to be trained, a secondary screening of the candidate sample dataset can be performed to obtain distilled training sample data for training the student model. Specifically, for each candidate sample data in the candidate sample dataset, the prediction difference between the student model and the fine-tuned teacher model can be determined, i.e., the distillation loss. Then, based on the distillation loss corresponding to each candidate sample data, its learning value can be determined. The distilled training samples are determined according to the magnitude of the learning value corresponding to each candidate sample data. In this way, a high-quality training sample set can be quickly constructed while reducing the inference computation overhead in the data screening process, thereby improving the efficiency of data screening and knowledge distillation, and helping to improve the knowledge distillation effect and the generalization ability of the trained model.

[0096] According to embodiments of this disclosure, this disclosure also provides an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to implement the knowledge distillation method described in any of the above embodiments when executed.

[0097] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions that enable a computer to implement the knowledge distillation method described in any of the above embodiments when executed.

[0098] According to embodiments of this disclosure, this disclosure also provides a computer program product including a computer program that, when executed by a processor, can implement the knowledge distillation method described in any of the above embodiments.

[0099] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein. In some alternative implementations of embodiments of the present disclosure, the electronic device 800 may be embodied as a server.

[0100] like Figure 8 As shown, the electronic device 800 includes a processing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0101] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0102] Processing unit 801 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processing unit 801 performs the various methods and processes described above, such as knowledge distillation methods. For example, in some embodiments, the knowledge distillation method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by processing unit 801, one or more steps of the update processing method described above may be performed. Alternatively, in other embodiments, processing unit 801 may be configured to perform knowledge distillation methods by any other suitable means (e.g., by means of firmware).

[0103] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0104] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0108] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology. Cloud servers, also known as cloud computing servers or cloud hosts, are a hosting product within the cloud computing service ecosystem, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak business scalability.

[0109] According to the technical solution of this disclosure, a small amount of sample data related to the target industry is used as prior information. Combined with the feature encoding layer of the teacher model after fine-tuning parameters, the sample data of the open domain dataset is coarsely screened to obtain a candidate sample dataset. Compared with the large-scale manual data selection and annotation of sample data construction methods, this method can reduce the data distribution difference between the data in the open domain dataset and the sample data related to the target industry, while reducing the acquisition cost of industry-related data and achieving low-cost and high-efficiency selection of industry-related data. Furthermore, based on the fine-tuned teacher model and the student model to be trained, a secondary screening of the candidate sample dataset can be performed to obtain distilled training sample data for training the student model. Specifically, for each candidate sample data in the candidate sample dataset, the prediction difference between the student model and the fine-tuned teacher model can be determined, i.e., the distillation loss. Then, based on the distillation loss corresponding to each candidate sample data, its learning value can be determined. The distilled training samples are determined according to the magnitude of the learning value corresponding to each candidate sample data. In this way, a high-quality training sample set can be quickly constructed while reducing the inference computation overhead in the data screening process, thereby improving the efficiency of data screening and knowledge distillation, and helping to improve the knowledge distillation effect and the generalization ability of the trained model.

[0110] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this disclosure can be achieved, and this is not limited herein.

[0111] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A knowledge distillation method, comprising: selecting, based on a sample dataset associated with a target industry and a feature encoding layer of a teacher model after fine-tuning, a candidate sample dataset from an open domain dataset, wherein a data quantity of the sample dataset associated with the target industry is less than a data quantity of the open domain dataset; determining, based on a student model and the teacher model after fine-tuning, a learning value degree of each candidate sample data in the candidate sample dataset for a distillation loss of the candidate sample data, and determining a distillation training sample dataset based on a size of each learning value degree; training the student model based on the distillation training sample dataset.

2. The method of claim 1, wherein, The selecting, based on a sample dataset associated with a target industry and a feature encoding layer of a teacher model after fine-tuning, a candidate sample dataset from an open domain dataset, comprises: determining, based on each sample data in the sample dataset associated with the target industry and the feature encoding layer of the teacher model after fine-tuning, a first embedding vector corresponding to the sample data; determining, based on each data in the open domain dataset and the feature encoding layer of the teacher model after fine-tuning, a second embedding vector corresponding to the data; determining, based on each second embedding vector and each first embedding vector, a similarity between each data in the open domain dataset and each sample data in the sample dataset associated with the target industry; selecting, based on a size of each similarity, the candidate sample dataset from the open domain dataset.

3. The method of claim 2, wherein, The selecting, based on a size of each similarity, the candidate sample dataset from the open domain dataset, comprises: selecting, for each data in the open domain dataset, a first sample data from the sample dataset associated with the target industry, a size of a similarity of which satisfies a first preset condition; determining, based on a similarity corresponding to each first sample data, a relevance score corresponding to each data in the open domain dataset; selecting, based on a size of a relevance score corresponding to each data in the open domain dataset, the candidate sample dataset from the open domain dataset.

4. The method of claim 3, wherein, The determining, based on a student model and the teacher model after fine-tuning, a learning value degree of each candidate sample data in the candidate sample dataset for a distillation loss of the candidate sample data, and determining a distillation training sample dataset based on a size of each learning value degree, comprises: determining, as a learning value degree of each candidate sample data, a result of weighted summation of a distillation loss corresponding to the candidate sample data and a relevance score; selecting, from the candidate sample dataset, a second sample data, a size of a learning value degree of which satisfies a second preset condition; determining the distillation training sample dataset based on the second sample data.

5. The method of claim 4, wherein, The determining the distillation training sample dataset based on the second sample data, comprises: combining the second sample data and a sample data in the sample dataset associated with the target industry, and determining a combined sample data as the distillation training sample dataset.

6. The method of claim 1, further comprising: determine a first loss of the student model for each of the candidate sample data; determine a second loss of the teacher model after the fine-tuning for each of the candidate sample data; determine a difference between the first loss and the second loss; determine the distillation loss based on the difference and a confidence weight corresponding to the teacher model after the fine-tuning.

7. The method of claim 6, wherein, The determination of the distillation loss based on the difference and the confidence weight corresponding to the teacher model after the fine-tuning includes: in response to determining that the difference is greater than a preset value, determining the distillation loss as a product of the difference and the confidence weight; or in response to determining that the difference is less than the preset value, determining the distillation loss as a product of the preset value and the confidence weight.

8. The method of claim 1, wherein, The training of the student model based on the distillation training sample data set includes: based on the distillation training sample data, determining a first output feature of the student model at a target layer and a second output feature of the teacher model after the fine-tuning at the target layer, wherein the target layer is determined based on a process of fine-tuning parameters of the teacher model; based on the first output feature and the second output feature, determining a feature alignment loss of the student model and the teacher model after the fine-tuning for the target layer; training the student model based on the feature alignment loss.

9. The method of claim 8, wherein, The training of the student model based on the feature alignment loss includes: based on a result of weighted summation of the feature alignment loss and a supervision loss corresponding to the student model, determining a total training loss; training the student model based on the total training loss.

10. The method of any one of claims 1 to 9, wherein, The sample data in the sample data set associated with the target industry includes multi-modal data associated with the target industry.

11. The method of claim 10, wherein, The sample data set associated with the target industry includes at least one of: a sample data set for medical diagnosis report generation, a sample data set for medical case retrieval, a sample data set for automatic driving scene recognition, a sample data set for traffic monitoring anomaly detection, a sample data set for traffic safety knowledge question answering, a sample data set for financial report analysis, a sample data set for risk control analysis, a sample data set for intelligent investment research question answering, a sample data set for contract review, a sample data set for judicial case retrieval, and a sample data set for legal regulation question answering.

12. A knowledge distillation apparatus, comprising: a first screening module configured to screen a candidate sample data set from an open domain data set based on a feature encoding layer of a teacher model after fine-tuning and a sample data set associated with a target industry, wherein a data amount of the sample data set associated with the target industry is less than a data amount of the open domain data set; a second screening module configured to determine a learning value degree of each of the candidate sample data based on a distillation loss of the student model and the teacher model after the fine-tuning for each of the candidate sample data in the candidate sample data set, and determine a distillation training sample data based on a size of each learning value degree. a knowledge distillation module configured to train the student model based on the distilled training sample data. 13.An electronic device comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the knowledge distillation method of any one of claims 1-11. 14.A non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the knowledge distillation method of any one of claims 1-11. 15.A computer program product comprising a computer program which, when executed by a processor, implements the steps of the knowledge distillation method according to any one of claims 1-11.

Images