Correlation search optimization method based on knowledge distillation technology and application system
Through knowledge distillation technology, the large BERT model is compressed into a lightweight model, which solves the problems of complex semantic understanding and model size in correlation search, and achieves efficient and accurate search results and rapid deployment.
Patent Information
- Application Number
- CN202410992798.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
Existing correlation search technologies are difficult to capture complex semantic relationships and contextual information, resulting in limited accuracy of search results, and the high complexity and large-scale model size of deep learning models such as CDSSM are not conducive to practical applications.
Using the relevance search optimization method based on knowledge distillation technology, the knowledge distillation of large pre-trained model BERT is used to migrate knowledge from the large pre-trained model BERT to a lighter-weight model, and the interactive model and representation model are used to reduce the model size and calculation requirements, while the model is used to reduce the model size and calculation requirements. Self-distillation technology compresses 12-layer large models into two-layer small models, maintaining high performance and improving online inference performance.
It achieves a significant reduction in model size and computing requirements while maintaining high performance, significantly improves search accuracy and efficiency, simplifies model adjustment and deployment process, enhances model availability and flexibility, and broadens application boundaries.
Smart Images

Figure CN118939758A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of distillation technology, and specifically relates to a correlation search optimization method and application system based on knowledge distillation technology. Background Art
[0002] Relevance search technology is in rapid iteration and innovation. From the traditional text matching-based method to the introduction of modern deep learning models, and then to the application of knowledge distillation technology, each step has a profound impact on the accuracy and efficiency of search. Traditional relevance search technologies, such as the vector space model (VSM) and the BM25 algorithm, although they have built a basic matching framework between documents and queries, are limited by the shallow level of semantic understanding and are difficult to capture complex semantic relationships and contextual information, resulting in limited accuracy of search results. With the rise of deep learning technology, deep learning models represented by DSSM and CDSSM have emerged in the field of relevance search. The DSSM model effectively encodes queries and documents through a dual-tower structure, achieves semantic-level matching, and significantly improves the accuracy and efficiency of search. The CDSSM model, on the other hand, takes advantage of the convolutional neural network to further explore the spatial features of the text. Although its model complexity is relatively high, it shows the great potential of deep models in relevance search. However, its model size is large and the training time is long, which is not conducive to its use in practical applications. In addition, as an effective means of model compression and acceleration, knowledge distillation technology is gradually penetrating into the field of natural language processing, including relevance search. Through knowledge distillation, the rich knowledge of large pre-trained models (such as BERT) can be effectively transferred to lighter models, such as Di still BERT, thereby significantly reducing the model size and computing requirements while maintaining high performance, which is of great significance for resource optimization and rapid deployment in practical applications. Summary of the invention
[0003] The purpose of the present invention is to provide a correlation search optimization method and application system based on knowledge distillation technology to solve the problems raised in the above background technology.
[0004] In order to achieve the above object, the present invention provides the following technical solution: a correlation search optimization method based on knowledge distillation technology, the optimization method steps are as follows:
[0005] S1, data preparation: Data preparation includes three parts: data collection, data cleaning and data labeling;
[0006] S2, interaction-based model: The interaction-based model inputs the input text into the BERT encoder to generate a text representation vector;
[0007] S3, representation-based model: The representation-based model inputs two input texts into two BERT encoders respectively to generate two text representation vectors;
[0008] S4, the interaction-based model obtains knowledge distillation based on the representation model: the representation-based model has become the mainstream technology for efficient text matching;
[0009] S5, interactive model self-distillation: pre-train a 12-layer interactive large model based on regular samples, and then use manually annotated samples to refine the interactive large model. Then, extract the first two layers from the 12-layer teacher model trained in the first step, and then use the corpus containing the teacher model knowledge obtained in the first step to pre-train these two layers of the model.
[0010] Preferably, the application system based on the above optimization method consists of a data acquisition module, a data cleaning module, and a data labeling module.
[0011] Preferably, the steps of the data preparation stage in S1 are as follows:
[0012] A1, data collection: according to the user's search exposure PV, data is collected by dividing it into high-frequency and long-tail to achieve data balance;
[0013] A2, data cleaning, after data collection is completed, the data needs to be cleaned;
[0014] A3, data labeling. According to the established correlation rules, the data is labeled according to four levels: 0, 1, 2, and 3. Level 0 indicates that the user's search query and the exposed doc are completely unrelated, level 1 indicates that the user's search query and the exposed doc are weakly correlated, level 2 indicates that the user's search query and the exposed doc are relatively correlated, and level 3 indicates that the user's search query and the exposed doc are completely correlated.
[0015] Preferably, the step of obtaining knowledge distillation based on the representation model from the interaction-based model in S4 is as follows:
[0016] B1, teacher model annotated corpus: first pre-train the interaction-based model based on rule samples, then use manually annotated samples to fine-tune the interaction-based model to obtain the teacher model, and then use the fine-tune teacher model to annotate large-scale corpora. The corpora annotated by the teacher model contain the knowledge learned by the teacher model itself;
[0017] B2, distilling knowledge to improve the accuracy of the representation model: Use the corpus annotated by the teacher model in the first step to pre-train the representation-based model. In this way, the knowledge learned by the interaction model is distilled to the representation-based model. Practice has shown that through this knowledge distillation method, the accuracy of the representation-based model is significantly improved.
[0018] Preferably, the interaction-based model self-distillation step in S5 is as follows:
[0019] C1, Distilled Large Model: Pre-train a 12-layer interactive large model based on rule samples, then use manually annotated samples to fine-tune the interactive large model to obtain a teacher model with higher accuracy, and then use the fine-tune teacher model to annotate large-scale corpora to obtain corpora containing teacher model knowledge;
[0020] C2, efficient learning with small models: The first two layers are extracted from the 12-layer teacher model obtained in the first step, and then the two-layer model is pre-trained using the corpus containing the teacher model knowledge obtained in the first step. Finally, the two-layer model is trained using manually labeled samples. Practice has shown that in this way, the final two-layer small model can basically learn all the knowledge of the 12-layer large model, with the accuracy basically the same as the 12-layer teacher model, and the online reasoning performance is improved by 6 to 10 times compared with the 12-layer teacher model.
[0021] Preferably, the data cleaning steps in A2 are as follows:
[0022] D1, delete duplicates: In large data sets, duplicate records can mislead the analysis results. By comparing the key fields of the records, these duplicates can be identified and deleted or merged to ensure the uniqueness of the data set;
[0023] D2, Correcting data errors: Data errors arise from a variety of reasons, such as input errors, equipment failures, or system problems. These include spelling errors, inconsistent formats, and logical errors. Correcting these errors requires careful review of the data and making adjustments based on business rules or common sense.
[0024] D3, fill missing values: Missing data is a common problem that affects the training effect of the model. Filling strategies include using the mean, median, mode, interpolation, predicting missing values with a prediction model, or setting default values based on business logic. The choice of strategy depends on the nature of the data and the proportion of missing data.
[0025] D4, Unified data format: Unified data format is the key to ensure data consistency, which includes the standardization of the format of date, time, currency, and numeric fields, as well as the cleaning of text data. In addition, data needs to be converted from one format to another to adapt to different analysis tools or systems;
[0026] D5, outlier handling: Outliers have an adverse effect on data analysis. Methods for identifying outliers include statistical methods and model-based methods. Outliers can be handled by deleting, replacing with normal values, or analyzing them separately.
[0027] D6, Normalized value: In machine learning and statistical modeling, it is often necessary to standardize or normalize numerical features to reduce the impact of different dimensions on model training;
[0028] D7, Integrate multiple sources: When data sets come from multiple sources, data integration is required, including field matching, data type unification, and merging operations. This requires a deep understanding of the structure and meaning of the data source to ensure the quality of the merged data;
[0029] D8, verify the cleaning effect: After completing the above steps, the cleaned data needs to be verified and tested to ensure that the cleaning process does not introduce new errors or omission of important information. This includes comparing the data before and after cleaning, checking the logical consistency of the data, and performing simple statistical analysis.
[0030] Preferably, the two methods of data annotation in A3 are as follows:
[0031] E1, rule-based automatic labeling method: The rule-based automatic labeling method uses pre-defined rules or models to automatically label data without manual intervention or with only a small amount of manual intervention. This method can significantly improve labeling efficiency and reduce labor costs, and is particularly suitable for processing large-scale data sets.
[0032] E2, manual labeling method: Manual labeling method refers to the process of manually reviewing and labeling data one by one. This method can ensure the accuracy and consistency of labeling, and is particularly suitable for complex and changeable labeling tasks.
[0033] Bert: a pre-trained model.
[0034] Query: the query statement used by the user when searching;
[0035] Doc: The document in the search results.
[0036] The beneficial effects of the present invention are as follows:
[0037] The present invention has demonstrated significant advantages in both the commercial and engineering fields through this innovative solution. From a commercial perspective, its core value lies in the high simplicity and replicability of the method, which can quickly achieve a significant increase in commercial value with a relatively low engineering cost investment. This solution is particularly suitable for building a lightweight correlation search system, accelerating the response speed and deployment cycle of applications, and helping companies seize the initiative in the highly competitive market. At the engineering level, this solution uses model compression technology to refine large and complex pre-trained models into compact and efficient small models, greatly reducing the computing burden and storage requirements, while maintaining or only slightly sacrificing model performance, achieving a leap in operating efficiency. This process not only simplifies the model adjustment and deployment process, but also enhances the availability and flexibility of the model, providing a solid foundation for rapid iteration and optimization. More importantly, the model extracted by distillation technology exhibits stronger generalization capabilities, which means that in different application scenarios such as speech recognition and image processing, it can show good adaptability and stability, further broadening the application boundaries and commercial value of the technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a schematic diagram of the steps of the optimization method of the present invention;
[0039] Figure 2 This is a schematic diagram of the interactive model structure of the present invention;
[0040] Figure 3 It is a schematic diagram of the model structure based on the representation of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] like Figures 1 to 3 As shown, the embodiment of the present invention provides a correlation search optimization method based on knowledge distillation technology, and the steps of the optimization method are as follows:
[0043] S1, data preparation: Data preparation includes three parts: data collection, data cleaning and data labeling;
[0044] S2, interaction-based model: The interaction-based model inputs the input text into the BERT encoder to generate a text representation vector;
[0045] S3, representation-based model: The representation-based model inputs two input texts into two BERT encoders respectively to generate two text representation vectors;
[0046] S4, the interaction-based model obtains knowledge distillation based on the representation model: the representation-based model has become the mainstream technology for efficient text matching;
[0047] S5, interactive model self-distillation: pre-train a 12-layer interactive large model based on regular samples, and then use manually annotated samples to refine the interactive large model. Then, extract the first two layers from the 12-layer teacher model trained in the first step, and then use the corpus containing the teacher model knowledge obtained in the first step to pre-train these two layers of the model.
[0048] Among them, the application system based on the above optimization method is composed of a data acquisition module, a data cleaning module, and a data labeling module.
[0049] The steps of data preparation in S1 are as follows:
[0050] A1, data collection: according to the user's search exposure PV, data is collected by dividing it into high-frequency and long-tail to achieve data balance;
[0051] A2, data cleaning, after data collection is completed, the data needs to be cleaned;
[0052] A3, data labeling. According to the established correlation rules, the data is labeled according to four levels: 0, 1, 2, and 3. Level 0 indicates that the user's search query and the exposed doc are completely unrelated, level 1 indicates that the user's search query and the exposed doc are weakly correlated, level 2 indicates that the user's search query and the exposed doc are relatively correlated, and level 3 indicates that the user's search query and the exposed doc are completely correlated.
[0053] The steps of knowledge distillation from the interaction-based model to the representation-based model in S4 are as follows:
[0054] B1, teacher model annotated corpus: first pre-train the interaction-based model based on rule samples, then use manually annotated samples to fine-tune the interaction-based model to obtain the teacher model, and then use the fine-tune teacher model to annotate large-scale corpora. The corpora annotated by the teacher model contain the knowledge learned by the teacher model itself;
[0055] B2, distilling knowledge to improve the accuracy of the representation model: Use the corpus annotated by the teacher model in the first step to pre-train the representation-based model. In this way, the knowledge learned by the interaction model is distilled to the representation-based model. Practice has shown that through this knowledge distillation method, the accuracy of the representation-based model is significantly improved.
[0056] Among them, the steps of interaction-based model self-distillation in S5 are as follows:
[0057] C1, Distilled Large Model: Pre-train a 12-layer interactive large model based on rule samples, then use manually annotated samples to fine-tune the interactive large model to obtain a teacher model with higher accuracy, and then use the fine-tune teacher model to annotate large-scale corpora to obtain corpora containing teacher model knowledge;
[0058] C2, efficient learning with small models: The first two layers are extracted from the 12-layer teacher model obtained in the first step, and then the two-layer model is pre-trained using the corpus containing the teacher model knowledge obtained in the first step. Finally, the two-layer model is trained using manually labeled samples. Practice has shown that in this way, the final two-layer small model can basically learn all the knowledge of the 12-layer large model, with the accuracy basically the same as the 12-layer teacher model, and the online reasoning performance is improved by 6 to 10 times compared with the 12-layer teacher model.
[0059] The data cleaning steps in A2 are as follows:
[0060] D1, delete duplicates: In large data sets, duplicate records can mislead the analysis results. By comparing the key fields of the records, these duplicates can be identified and deleted or merged to ensure the uniqueness of the data set;
[0061] D2, Correcting data errors: Data errors arise from a variety of reasons, such as input errors, equipment failures, or system problems. These include spelling errors, inconsistent formats, and logical errors. Correcting these errors requires careful review of the data and making adjustments based on business rules or common sense.
[0062] D3, fill missing values: Missing data is a common problem that affects the training effect of the model. Filling strategies include using the mean, median, mode, interpolation, predicting missing values with a prediction model, or setting default values based on business logic. The choice of strategy depends on the nature of the data and the proportion of missing data.
[0063] D4, Unified data format: Unified data format is the key to ensure data consistency, which includes the standardization of the format of date, time, currency, and numeric fields, as well as the cleaning of text data. In addition, data needs to be converted from one format to another to adapt to different analysis tools or systems;
[0064] D5, outlier handling: Outliers have an adverse effect on data analysis. Methods for identifying outliers include statistical methods and model-based methods. Outliers can be handled by deleting, replacing with normal values, or analyzing them separately.
[0065] D6, Normalized value: In machine learning and statistical modeling, it is often necessary to standardize or normalize numerical features to reduce the impact of different dimensions on model training;
[0066] D7, Integrate multiple sources: When data sets come from multiple sources, data integration is required, including field matching, data type unification, and merging operations. This requires a deep understanding of the structure and meaning of the data source to ensure the quality of the merged data;
[0067] D8, verify the cleaning effect: After completing the above steps, the cleaned data needs to be verified and tested to ensure that the cleaning process does not introduce new errors or omission of important information. This includes comparing the data before and after cleaning, checking the logical consistency of the data, and performing simple statistical analysis.
[0068] The two methods of data labeling in A3 are as follows:
[0069] E1, rule-based automatic labeling method: The rule-based automatic labeling method uses pre-defined rules or models to automatically label data without manual intervention or with only a small amount of manual intervention. This method can significantly improve labeling efficiency and reduce labor costs, and is particularly suitable for processing large-scale data sets.
[0070] E2, manual labeling method: Manual labeling method refers to the process of manually reviewing and labeling data one by one. This method can ensure the accuracy and consistency of labeling, and is particularly suitable for complex and changeable labeling tasks.
[0071] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A relevance search optimization method based on knowledge distillation technology, Features: The optimization method steps are as follows: S1, data preparation: Data preparation includes three parts: data collection, data cleaning and data labeling; S2, interaction-based model: The interaction-based model inputs the input text into the BERT encoder to generate a text representation vector; S3, representation-based model: The representation-based model inputs two input texts into two BERT encoders respectively to generate two text representation vectors; S4, the interaction-based model obtains knowledge distillation based on the representation model: the representation-based model has become the mainstream technology for efficient text matching; S5, interactive model self-distillation: pre-train a 12-layer interactive large model based on regular samples, and then fine-tune the interactive large model using manually annotated samples. Subsequently, the first two layers are extracted from the 12-layer teacher model trained in the first step, and then the two-layer model is pre-trained using the corpus containing the teacher model knowledge obtained in the first step.
2. According to claim 1, a correlation search optimization method based on knowledge distillation technology, Features: The steps of the data preparation phase in S1 are as follows: A1, data collection: according to the user's search exposure PV, data is collected by dividing it into high-frequency and long-tail to achieve data balance; A2, data cleaning, after data collection is completed, the data needs to be cleaned; A3, data labeling. According to the established correlation rules, the data is labeled according to four levels: 0, 1, 2, and 3. Level 0 indicates that the user's search query and the exposed doc are completely unrelated, level 1 indicates that the user's search query and the exposed doc are weakly correlated, level 2 indicates that the user's search query and the exposed doc are relatively correlated, and level 3 indicates that the user's search query and the exposed doc are completely correlated.
3. According to claim 1, a correlation search optimization method based on knowledge distillation technology, Features: The steps of obtaining knowledge distillation based on the representation model from the interaction-based model in S4 are as follows: B1, teacher model annotated corpus: First, the interaction-based model is pre-trained based on rule samples, and then the interaction-based model is fine-tuned using manually annotated samples to obtain the teacher model. The fine-tuned teacher model is then used to annotate large-scale corpora. The corpora annotated by the teacher model contain the knowledge learned by the teacher model itself. B2, distilling knowledge to improve the accuracy of the representation model: Use the corpus annotated by the teacher model in the first step to pre-train the representation-based model. In this way, the knowledge learned by the interaction model is distilled to the representation-based model. Practice has shown that through this knowledge distillation method, the accuracy of the representation-based model is significantly improved.
4. According to claim 1, a correlation search optimization method based on knowledge distillation technology, Features: The steps of the interaction-based model self-distillation in S5 are as follows: C1, Distillation Large Model: Pre-train a 12-layer interactive large model based on rule samples, then fine-tune the interactive large model using manually annotated samples to obtain a teacher model with higher accuracy. Then use the fine-tuned teacher model to annotate a large-scale corpus to obtain a corpus containing the teacher model knowledge. C2, small model efficient learning: extract the first two layers from the 12-layer teacher model obtained in the first step, then use the corpus containing the teacher model knowledge obtained in the first step to pre-train the two-layer model, and finally use manually labeled samples to fine-tune the two-layer model. Practice has proved that in this way, the final two-layer small model can basically learn all the knowledge of the 12-layer large model, and the accuracy is basically the same as the 12-layer teacher model. The online reasoning performance is improved by 6 to 10 times compared with the 12-layer teacher model.
5. According to claim 2, a correlation search optimization method based on knowledge distillation technology, Features: The data cleaning steps in A2 are as follows: D1, delete duplicates: In large data sets, duplicate records can mislead the analysis results. By comparing the key fields of the records, these duplicates can be identified and deleted or merged to ensure the uniqueness of the data set; D2, correcting data errors: Data errors come from a variety of reasons, such as input errors, equipment failures, or system problems. This includes spelling errors, inconsistent formats, and logical errors. Correcting these errors requires careful review of the data and making adjustments based on business rules or common sense. D3, filling missing values: Missing data is a common problem that affects the training effect of the model. Filling strategies include using the mean, median, mode, interpolation, predicting missing values with a prediction model, or setting default values based on business logic. The choice of strategy depends on the nature of the data and the proportion of missing data. D4, Unified Data Format: Unified data format is the key to ensuring data consistency, which includes format standardization of date, time, currency, and numeric fields, as well as cleaning of text data. In addition, data needs to be converted from one format to another to adapt to different analysis tools or systems; D5, outlier handling: Outliers have an adverse effect on data analysis. Methods for identifying outliers include statistical methods and model-based methods. Outliers can be handled by deleting, replacing with normal values, or analyzing them separately. D6, Normalized value: In machine learning and statistical modeling, it is often necessary to standardize or normalize numerical features to reduce the impact of different dimensions on model training; D7, Integrate multiple sources: When data sets come from multiple sources, data integration is required, including field matching, data type unification, and merging operations. This requires a deep understanding of the structure and meaning of the data source to ensure the quality of the merged data. D8, verify the cleaning effect: After completing the above steps, the cleaned data needs to be verified and tested to ensure that the cleaning process does not introduce new errors or omission of important information. This includes comparing the data before and after cleaning, checking the logical consistency of the data, and performing simple statistical analysis.
6. A correlation search optimization method based on knowledge distillation technology according to claim 2, Features: The two methods of data annotation in A3 are as follows: E1, rule-based automatic labeling method: The rule-based automatic labeling method uses pre-defined rules or models to automatically label data without manual intervention or with only a small amount of manual intervention. This method can significantly improve labeling efficiency and reduce labor costs, and is particularly suitable for processing large-scale data sets. E2, manual labeling method: Manual labeling method refers to the process of manually reviewing and labeling data one by one. This method can ensure the accuracy and consistency of labeling, and is particularly suitable for complex and changeable labeling tasks.
7. An application system of the correlation search optimization method based on knowledge distillation technology according to any one of claims 1 to 6, Features: The application system consists of a data acquisition module, a data cleaning module, and a data labeling module.
Citation Information
Patent Citations
Text intention classification method and device and readable medium
CN112905795A
Text matching method and device and electronic equipment
CN114780709A
Knowledge distillation method, device and program product of text matching double-tower model
CN114841172A
Correlation judgment method and LLM-based correlation judgment model construction method
CN117743950A
Resource retrieval method and device, training method and device, electronic equipment, storage medium and program product
CN117851546A
