Duplicate checking small language model training method combined with multi-level knowledge distillation

Abstract

The embodiment of the invention provides a duplicate checking small language model training method combined with multilevel knowledge distillation, and the method comprises the steps: obtaining duplicate checking sample pairs, and determining the complexity of the duplicate checking sample pairs according to the text features of the duplicate checking sample pairs; according to the complexity of the duplicate checking sample pair, determining distillation levels of the teacher model, and determining a weighting coefficient of each distillation level of the teacher model; determining the distillation loss between the teacher model and the student model according to the weighting coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model and the second output result of each distillation level of the student model; according to the distillation loss between the teacher model and the student model, updating parameters of the student model; and repeating the above steps until the updated student model meets the preset condition, and taking the updated student model as a duplicate checking small language model, thereby realizing a low-power-consumption and high-precision duplicate checking effect.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence algorithm technology, and in particular to a method, device, electronic device and storage medium for training a small language model for duplicate detection combined with multi-level knowledge distillation. Background Art

[0002] Plagiarism detection models are AI systems used to detect text similarity or plagiarism. They are widely used in areas such as academic paper plagiarism detection, assignment grading, news originality review, and legal document comparison. Existing plagiarism detection models are typically generated based on general-purpose large language models. Large language models are large and complex models. While they possess excellent text processing capabilities, they require advanced hardware to meet computing resource requirements. For well-defined application scenarios, such as text plagiarism detection, using general-purpose large language processing models can waste model capabilities and overconsume resources. Summary of the Invention

[0003] In order to solve the above problems, the embodiments of the present invention disclose a method, device, electronic device and storage medium for training a small language model for duplicate detection combined with multi-level knowledge distillation.

[0004] In a first aspect, an embodiment of the present invention provides a method for training a small language model for duplicate detection combined with multi-level knowledge distillation, the method comprising:

[0005] Obtaining a duplicate checking sample pair, and determining the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair;

[0006] Determining a distillation level of a teacher model according to the complexity of the duplicate checking sample pair, and determining a weighting coefficient of each distillation level of the teacher model;

[0007] Input the duplicate checking sample pair into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate checking sample pair into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model;

[0008] Determining a distillation loss between the teacher model and the student model according to a weighted coefficient of each network distillation of the teacher model, a first output result of each distillation level of the teacher model, and a second output result of each distillation level of the student model;

[0009] updating the parameters of the student model according to the distillation loss between the teacher model and the student model;

[0010] Repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as the small language model for checking for duplicates.

[0011] Optionally, the text features of the duplicate checking sample pair include text length, lexical diversity, and semantic density, and determining the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair includes:

[0012] Obtaining preset weight coefficients corresponding to the text length, the lexical diversity, and the semantic density;

[0013] Normalizing the text length, lexical diversity, and semantic density of the duplicate checking sample pairs respectively to obtain normalized scores of the text length, lexical diversity, and semantic density of the duplicate checking sample pairs;

[0014] According to the preset weight coefficients corresponding to the text length, the lexical diversity and the semantic density, the normalized scores of the text length, lexical diversity and semantic density of the duplicate checking sample pair are weighted and summed to obtain the complexity of the duplicate checking sample pair.

[0015] Optionally, determining the distillation level of the teacher model according to the complexity of the duplicate checking sample pair includes:

[0016] When the complexity of the duplicate checking sample pair is greater than a preset threshold, the first half of the network layer of the teacher model is used as the distillation layer of the teacher model;

[0017] When the complexity of the duplicate check sample pair is less than or equal to the preset threshold, the second half of the network layer of the teacher model is used as the distillation layer of the teacher model.

[0018] Optionally, determining a weight coefficient of each distillation level of the teacher model includes:

[0019] The weight coefficient of each distillation level of the teacher model is determined according to the following formula:

[0020]

[0021] Among them, l is the current distillation level, fl(C(x)) is the weight coefficient of the current distillation level, L is the total network level of the teacher model, and C(x) is the complexity of the duplicate checking sample pair.

[0022] Optionally, determining the distillation loss between the teacher model and the student model according to the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model includes:

[0023] Determining a correspondence between each distillation level of the teacher model and each distillation level of the student model according to the number of distillation levels of the teacher model and the number of distillation levels of the student model;

[0024] Determining, according to a correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model;

[0025] According to the weighted coefficient of each network layer of the teacher model, the hierarchical distillation loss between the teacher model and the student model is weightedly summed to obtain the distillation loss between the teacher model and the student model.

[0026] Optionally, determining, based on the correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of a corresponding distillation level of the student model includes:

[0027] According to the following formula, based on the corresponding relationship between each distillation level of the teacher model and each distillation level of the student model, the hierarchical distillation loss between the first output result of each distillation level of the teacher model and the second output result of the corresponding distillation level of the student model is determined:

[0028] L l (x) = KL(S l′ (x)||T l (x))

[0029] Among them, l is the current distillation level of the teacher model, l′ is the distillation level of the student model corresponding to the current distillation level of the teacher model, and L l (x) is the distillation loss, S l′ (x) is the second output result of the distillation layer of the student model, T l (x) is the first output result of the distillation layer of the teacher model, KL(P||Q)=∑ i P(i)log(P(i) / Q(i)).

[0030] Optionally, updating the parameters of the student model according to the distillation loss between the teacher model and the student model includes:

[0031] Get the preset learning rate;

[0032] Determining a parameter gradient of the student model according to the distillation loss between the teacher model and the student model;

[0033] The parameters of the student model are updated according to the preset learning rate and the parameter gradient of the student model.

[0034] Optionally, determining a parameter gradient of the student model according to a distillation loss between the teacher model and the student model includes:

[0035] The parameter gradient of the student model is determined according to the distillation loss between the teacher model and the student model according to the following formula:

[0036]

[0037] Among them, W is the weight matrix, L is the total network level of the teacher model, l is the current distillation level, α l is the weight coefficient of the current distillation level, L distill is the distillation loss between the teacher model and the student model;

[0038] The updating of the parameters of the student model according to the preset learning rate and the parameter gradient of the student model includes:

[0039] According to the following formula, the parameters of the student model are updated according to the preset learning rate and the parameter gradient of the student model:

[0040]

[0041] Among them, θ′ studengt is the parameter of the updated student model, θ student is the parameter of the student model before updating, η is the preset learning rate, is the parameter gradient of the student model.

[0042] Optionally, the above steps are repeated until the updated student model meets the preset conditions, and the updated student model is used as the small language model for checking duplicates, including:

[0043] Repeat the above steps until the distillation loss between the teacher model and the student model does not decrease within a preset distillation period, then determine that the updated student model meets the preset condition;

[0044] The updated student model is used as a small language model for checking for duplicate content.

[0045] In a second aspect, an embodiment of the present invention provides a small language model training device for duplicate detection combined with multi-level knowledge distillation, the device comprising:

[0046] A text complexity determination module is used to obtain a duplicate check sample pair and determine the complexity of the duplicate check sample pair based on the text features of the duplicate check sample pair;

[0047] A text complexity utilization module, configured to determine a distillation level of a teacher model according to the complexity of the duplicate check sample pair, and to determine a weighting coefficient of each distillation level of the teacher model;

[0048] A model result output module is configured to input the duplicate check sample pair into the teacher model to obtain a first output result of each distillation level of the teacher model; and input the duplicate check sample pair into the student model to obtain a second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model;

[0049] a distillation loss determination module, configured to determine the distillation loss between the teacher model and the student model based on the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model;

[0050] A student model parameter updating module, configured to update the parameters of the student model according to the distillation loss between the teacher model and the student model;

[0051] The duplicate checking language model generation module is used to repeat the above steps until the updated student model meets the preset conditions, and then the updated student model is used as the duplicate checking small language model.

[0052] Optionally, the text features of the duplicate checking sample pair include text length, lexical diversity and semantic density, and the text complexity determination module includes:

[0053] A preset weight coefficient acquisition submodule is used to obtain preset weight coefficients corresponding to the text length, the lexical diversity and the semantic density;

[0054] A normalized score determination submodule is used to normalize the text length, lexical diversity and semantic density of the duplicate check sample pair respectively to obtain normalized scores of the text length, lexical diversity and semantic density of the duplicate check sample pair;

[0055] The text complexity determination submodule is used to perform weighted summation on the normalized scores of the text length, lexical diversity and semantic density of the duplicate checking sample pair according to the preset weight coefficients corresponding to the text length, lexical diversity and semantic density, so as to obtain the complexity of the duplicate checking sample pair.

[0056] Optionally, the text complexity using module includes:

[0057] A first distillation level determination submodule is configured to use the first half of the network level of the teacher model as the distillation level of the teacher model when the complexity of the duplicate check sample pair is greater than a preset threshold;

[0058] The second distillation level determination submodule is used to use the second half of the network level of the teacher model as the distillation level of the teacher model when the complexity of the duplicate check sample pair is less than or equal to the preset threshold.

[0059] Optionally, the text complexity usage module further includes:

[0060] The weight coefficient determination submodule is used to determine the weight coefficient of each distillation level of the teacher model according to the following formula:

[0061]

[0062] Among them, l is the current distillation level, fl(C(x)) is the weight coefficient of the current distillation level, L is the total network level of the teacher model, and C(x) is the complexity of the duplicate checking sample pair.

[0063] Optionally, the distillation loss determination module includes:

[0064] a distillation level correspondence determination submodule, configured to determine a correspondence between each distillation level of the teacher model and each distillation level of the student model based on the number of distillation levels of the teacher model and the number of distillation levels of the student model;

[0065] a hierarchical distillation loss determination submodule, configured to determine, based on a correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model;

[0066] The distillation loss determination submodule is used to perform weighted summation of the hierarchical distillation losses between the teacher model and the student model according to the weight coefficient of each network layer of the teacher model to obtain the distillation loss between the teacher model and the student model.

[0067] Optionally, the hierarchical distillation loss determination submodule includes:

[0068] A hierarchical distillation loss determining unit is configured to determine, according to the following formula, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model based on a correspondence between each distillation level of the teacher model and each distillation level of the student model:

[0069] L l (x) = KL(S l′ (x)||T l (x))

[0070] Among them, l is the current distillation level of the teacher model, l′ is the distillation level of the student model corresponding to the current distillation level of the teacher model, and L l (x) is the distillation loss, S l′ (x) is the second output result of the distillation layer of the student model, T l (x) is the first output result of the distillation layer of the teacher model, KL(P||Q)=∑ i P(i)log(P(i) / Q(i)).

[0071] Optionally, the student model parameter updating module includes:

[0072] The preset learning rate acquisition submodule is used to obtain the preset learning rate;

[0073] a parameter gradient determination submodule, configured to determine a parameter gradient of the student model based on the distillation loss between the teacher model and the student model;

[0074] The student model parameter updating submodule is used to update the parameters of the student model according to the preset learning rate and the parameter gradient of the student model.

[0075] Optionally, the parameter gradient determination submodule includes:

[0076] A parameter gradient determination unit is configured to determine the parameter gradient of the student model according to the distillation loss between the teacher model and the student model according to the following formula:

[0077]

[0078] Among them, W is the weight matrix, L is the total network level of the teacher model, l is the current distillation level, α l is the weight coefficient of the current distillation level, L distill is the distillation loss between the teacher model and the student model;

[0079] The student model parameter updating submodule includes:

[0080] The student model parameter updating unit is used to update the parameters of the student model according to the preset learning rate and the parameter gradient of the student model according to the following formula:

[0081]

[0082] Among them, θ′ studengt is the parameter of the updated student model, θ student is the parameter of the student model before updating, η is the preset learning rate, is the parameter gradient of the student model.

[0083] Optionally, the duplicate checking language model generation module includes:

[0084] A preset condition judgment submodule is used to repeat the above steps until the distillation loss between the teacher model and the student model does not decrease within a preset distillation period, and then determine that the updated student model meets the preset condition;

[0085] The duplicate checking language model generation submodule is used to use the updated student model as a duplicate checking small language model.

[0086] In the third aspect, the present invention shows an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the computer program, it implements the steps of the above-mentioned method for training a small language model for duplicate detection combined with multi-level knowledge distillation.

[0087] In a fourth aspect, the present invention shows a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for training a small language model for duplicate detection combined with multi-level knowledge distillation are implemented.

[0088] The embodiments of the present invention include the following advantages:

[0089] The embodiment of the present invention can obtain duplicate detection sample pairs, and determine the complexity of the duplicate detection sample pairs according to the text features of the duplicate detection sample pairs, thereby overcoming the limitations of traditional duplicate detection methods in evaluating text complexity by analyzing text features; determine the distillation level of the teacher model according to the complexity of the duplicate detection sample pairs, and determine the weighting coefficient of each distillation level of the teacher model, so as to dynamically adjust the allocation of computing resources and improve the distillation efficiency by dynamically associating the text complexity with the knowledge distillation level; input the duplicate detection sample pairs into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate detection sample pairs into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model; according to each level of the teacher model The weighted coefficients of network distillation, the first output results of each distillation layer of the teacher model, and the second output results of each distillation layer of the student model are used to determine the distillation loss between the teacher model and the student model, thereby improving the effect of knowledge distillation in the text duplication detection task through multi-level distillation loss calculation; according to the distillation loss between the teacher model and the student model, the parameters of the student model are updated, thereby optimizing the student model through dynamic distillation and loss feedback; repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as a small language model for duplication detection, so that the student model can gradually absorb the deep matching ability of the teacher model. At the same time, the small language model can avoid excessive consumption of resources and achieve low-power, high-precision duplication detection effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0091] Figure 1 This is a flowchart of the steps of a method for training a small language model for duplicate checking combined with multi-level knowledge distillation according to an embodiment of the present invention;

[0092] Figure 2 This is a flowchart of another method for training a small language model for duplicate checking combined with multi-level knowledge distillation according to an embodiment of the present invention;

[0093] Figure 3 This is a logic diagram of a method for training a small language model for duplicate checking combined with multi-level knowledge distillation according to an embodiment of the present invention;

[0094] Figure 4 This is a structural block diagram of a small language model training device for checking for duplicates combined with multi-level knowledge distillation in an embodiment of the present invention. DETAILED DESCRIPTION

[0095] The present invention proposes a method for training a small language model for duplicate detection that combines multi-level knowledge distillation, with the aim of reducing the excessive consumption of resources by the duplicate detection model while ensuring the quality of text duplicate detection. To achieve this goal, an embodiment of the present invention determines the complexity of the duplicate detection sample pair based on the text features of the duplicate detection sample pair; determines the distillation level of the teacher model and the weighting coefficient of each distillation level based on the complexity of the duplicate detection sample pair; determines the distillation loss between the teacher model and the student model based on the weighting coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model; and updates the parameters of the student model based on the distillation loss, and uses the updated student model as a small language model for duplicate detection, so that the student model can gradually absorb the deep matching ability of the teacher model while avoiding excessive consumption of resources, thereby achieving a low-power, high-precision duplicate detection effect.

[0096] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0097] Reference Figure 1 , shows a flowchart of the steps of a method for training a small language model for duplicate checking combined with multi-level knowledge distillation according to an embodiment of the present invention. The method may specifically include the following steps:

[0098] Step 101: obtaining a duplicate check sample pair and determining the complexity of the duplicate check sample pair based on the text features of the duplicate check sample pair;

[0099] In an embodiment of the present invention, duplicate checking sample pairs can be obtained through the following ways: academic databases, such as duplicate checking system samples provided by HowNet, Wanfang, and VIP; public data sets, such as the plagiarism detection data set of the Pyramid Network evaluation task; self-built sample pairs, that is, samples of different similarities are created by manually rewriting text; educational institution resources, such as the duplicate checking case library that may be provided by universities or research institutions.

[0100] The present invention can determine the complexity of a duplicate sample pair based on its textual features, thereby overcoming the limitations of traditional methods for assessing text complexity through textual feature analysis. Furthermore, it can employ different detection strategies for plagiarism behaviors of varying complexity, improving detection efficiency. Furthermore, it can distinguish legitimate citations from malicious plagiarism, reducing the error rate in duplicate detection. Through systematic sample acquisition and scientific complexity assessment, it achieves a leap from simple duplication detection to intelligent plagiarism identification.

[0101] Step 102: determining a distillation level of a teacher model according to the complexity of the duplicate check sample pair, and determining a weighting coefficient of each distillation level of the teacher model;

[0102] In an embodiment of the present invention, the distillation level of the teacher model can be determined according to the complexity of the duplicate checking sample pairs, and the allocation of computing resources can be adjusted dynamically. For high-complexity texts, a shallow network can be distilled to capture basic features such as vocabulary and grammar to handle complex rewritings that require fine-grained comparison; for low-complexity texts, a deep network can be distilled to encode semantic and paragraph structure features to handle texts that are similar on the surface but simple in semantics. The embodiment of the present invention extracts features of different levels from multiple levels of the teacher model through multi-level knowledge distillation, so that the student model can learn deeper text semantics and structural information, thereby enhancing the model's ability to understand complex texts; at the same time, by selecting levels based on text complexity, full-level calculations can be avoided, thereby improving the overall distillation efficiency.

[0103] This embodiment of the present invention also dynamically adjusts the distillation weights at different levels based on the complexity of the duplicate-checking sample pairs. For simple text, higher-level distillation weights are increased, thereby reducing unnecessary computational resource consumption. For complex text, lower-level distillation weights are increased to ensure the model fully captures the text's details. Setting the weighted coefficients at each distillation level effectively optimizes computational efficiency, improving the model's inference speed and response time while maintaining accurate duplicate-checking results.

[0104] Step 103: Input the duplicate check sample pair into the teacher model to obtain a first output result of each distillation level of the teacher model; and input the duplicate check sample pair into the student model to obtain a second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model;

[0105] In an embodiment of the present invention, the teacher model can be a large model of BERT-large (24 layers) or RoBERTa (16 layers), and the student model can be a lightweight model of DistilBERT (6 layers) or TinyBERT (4 layers). Among them, the low layers of the teacher model, such as layers 1-6, are used to process vocabulary overlap; the middle layers of the teacher model, such as layers 7-12, are used to capture sentence structure; and the high layers of the teacher model, such as layers 13+, are used to characterize semantic similarity. In an embodiment of the present invention, the duplicate checking sample pairs can be input into two models at the same time to obtain the first output result of each distillation level of the teacher model and the second output result of each distillation level of the student model, so as to compress the multi-layer features of the teacher into the single-layer parameters of the student model in the future. Through the refined hierarchical knowledge transfer, the student model can reduce the consumption of computing resources while improving the accuracy of duplicate checking.

[0106] Step 104: determining a distillation loss between the teacher model and the student model based on the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model;

[0107] In an embodiment of the present invention, KL divergence (Kullback-Leibler Divergence) can be used to measure the similarity between the student model and the teacher model, that is, the distillation loss between the teacher model and the student model. During the calculation process, not only the KL divergence of the final output is obtained, but also each distillation level of the student model and the teacher model can be compared and learned to enhance the matching degree of the feature space. Since different distillation levels contribute differently to the final task, dynamic weighted distillation loss can be used to ensure that the knowledge of the key levels is transferred first. The embodiment of the present invention significantly improves the effect of knowledge distillation in text similarity tasks through multi-level distillation loss calculation.

[0108] Step 105: updating the parameters of the student model according to the distillation loss between the teacher model and the student model;

[0109] In an embodiment of the present invention, the student model is optimized through the distillation loss feedback between the teacher model and the student model, which can improve the accuracy of the student model in checking for duplicate content in complex texts, such as academic papers and scientific research project reports, and reduce missed detections and false detections.

[0110] Step 106, repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as the small language model for checking for duplicates.

[0111] In an embodiment of the present invention, through multiple rounds of loss feedback updates, the student model can gradually absorb the deep matching capabilities of the teacher model. Since the student model is a small language model and is combined with knowledge distillation technology, the resulting small language model for duplicate detection can run efficiently on resource-constrained devices (such as terminal devices). By reducing dependence on server-side computing resources, the small language model for duplicate detection can effectively reduce hardware resource consumption and reduce operating costs.

[0112] The embodiment of the present invention can obtain duplicate detection sample pairs, and determine the complexity of the duplicate detection sample pairs according to the text features of the duplicate detection sample pairs, thereby overcoming the limitations of traditional duplicate detection methods in evaluating text complexity by analyzing text features; determine the distillation level of the teacher model according to the complexity of the duplicate detection sample pairs, and determine the weighting coefficient of each distillation level of the teacher model, so as to dynamically adjust the allocation of computing resources and improve the distillation efficiency by dynamically associating the text complexity with the knowledge distillation level; input the duplicate detection sample pairs into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate detection sample pairs into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model; according to each level of the teacher model The weighted coefficients of network distillation, the first output results of each distillation layer of the teacher model, and the second output results of each distillation layer of the student model are used to determine the distillation loss between the teacher model and the student model, thereby improving the effect of knowledge distillation in the text duplication detection task through multi-level distillation loss calculation; according to the distillation loss between the teacher model and the student model, the parameters of the student model are updated, thereby optimizing the student model through dynamic distillation and loss feedback; repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as a small language model for duplication detection, so that the student model can gradually absorb the deep matching ability of the teacher model. At the same time, the small language model can avoid excessive consumption of resources and achieve low-power, high-precision duplication detection effects.

[0113] Reference Figure 2 , shows a flowchart of another method for training a small language model for duplicate checking combined with multi-level knowledge distillation according to an embodiment of the present invention. The method may specifically include the following steps:

[0114] Step 201: obtaining a duplicate check sample pair and determining the complexity of the duplicate check sample pair based on the text features of the duplicate check sample pair;

[0115] In one embodiment, the text features of the duplicate checking sample pair include text length, lexical diversity, and semantic density. The step of determining the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair may further include the following sub-steps:

[0116] Sub-step S11, obtaining preset weight coefficients corresponding to the text length, the lexical diversity, and the semantic density;

[0117] Sub-step S12, normalizing the text length, lexical diversity, and semantic density of the duplicate check sample pair, respectively, to obtain normalized scores of the text length, lexical diversity, and semantic density of the duplicate check sample pair;

[0118] Sub-step S13, according to the preset weight coefficients corresponding to the text length, the lexical diversity and the semantic density, the normalized scores of the text length, lexical diversity and semantic density of the duplicate checking sample pair are weighted and summed to obtain the complexity of the duplicate checking sample pair.

[0119] In an embodiment of the present invention, the complexity of the duplicate checking sample pair can be obtained by weighting and summing the normalized scores of the text length, lexical diversity, and semantic density of the duplicate checking sample pair according to the preset weight coefficients corresponding to the text length, lexical diversity, and semantic density of the duplicate checking sample pair. The summation formula is as follows:

[0120] c=λ1*flen(x)+λ2*fvocab(x)+λ3*fsem(x)

[0121] Where c is the complexity of the duplicate checking sample pair, flen(x) represents the score after normalization of text length, fvocab(x) represents the score after normalization of lexical diversity, and fsem(x) represents semantic density. The scores of text length, lexical diversity, and semantic density are all normalized to [0, 1]. λ1, λ2, and λ3 are the weights of each indicator, which can be adjusted according to actual scenarios and experience.

[0122] The present invention can determine the complexity of a duplicate sample pair based on its textual features, thereby overcoming the limitations of traditional methods for assessing text complexity through textual feature analysis. Furthermore, it can employ different detection strategies for plagiarism behaviors of varying complexity, improving detection efficiency. Furthermore, it can distinguish legitimate citations from malicious plagiarism, reducing the error rate in duplicate detection. Through systematic sample acquisition and scientific complexity assessment, it achieves a leap from simple duplication detection to intelligent plagiarism identification.

[0123] Step 202: determining a distillation level of the teacher model according to the complexity of the duplicate check sample pair, and determining a weighting coefficient of each distillation level of the teacher model;

[0124] In one embodiment, the step of determining the distillation level of the teacher model according to the complexity of the duplicate check sample pair may further include the following sub-steps:

[0125] Sub-step S21, when the complexity of the duplicate check sample pair is greater than a preset threshold, using the first half of the network layer of the teacher model as the distillation layer of the teacher model;

[0126] Sub-step S22, when the complexity of the duplicate check sample pair is less than or equal to the preset threshold, the latter half of the network layer of the teacher model is used as the distillation layer of the teacher model.

[0127] In this embodiment of the present invention, the distillation level of the teacher model can be determined based on the complexity of the duplicate sample pairs, dynamically adjusting the allocation of computing resources. For highly complex texts, a shallow network can be distilled to capture basic features such as vocabulary and grammar to handle complex rewrites that require fine-grained comparison. For low-complexity texts, a deep network can be distilled to encode semantic and text structure features to handle texts that are superficially similar but semantically simple.

[0128] For example, assuming that the complexity evaluation range is [0, 1] and the preset threshold is 0.5, when the complexity of the duplicate checking sample pair c>0.5, the first half of the network level of the teacher model [1, L / 2] distilled knowledge can be selected. When the complexity of the duplicate checking sample pair c<=0.5, the second half of the network level of the teacher model [L / 2+1, L] distilled knowledge can be selected, where L is the total network level of the teacher model. Those skilled in the art can set the preset threshold to other appropriate values ​​based on the idea of ​​the present invention, and the present invention is not limited to this.

[0129] The embodiment of the present invention uses multi-level knowledge distillation to extract features at different levels from multiple levels of the teacher model, enabling the student model to learn deeper text semantics and structural information, thereby enhancing the model's ability to understand complex texts. At the same time, by selecting levels based on text complexity, full-level calculations can be avoided, thereby improving overall distillation efficiency.

[0130] In one embodiment, the step of determining the weight coefficient of each distillation level of the teacher model may further include the following sub-steps:

[0131] Sub-step S31, determining the weight coefficient of each distillation level of the teacher model according to the following formula:

[0132]

[0133] Among them, l is the current distillation level, fl(C(x)) is the weight coefficient of the current distillation level, L is the total network level of the teacher model, and C(x) is the complexity of the duplicate checking sample pair.

[0134] In an embodiment of the present invention, the distillation weights at different levels can be dynamically adjusted based on the complexity of the duplicate check sample pairs. Specifically, the weight coefficients of each level can be calculated using a nonlinear function. For texts with lower complexity, such as short articles, a decreasing function can be used to increase the distillation weights at higher levels and lower levels. For texts with higher complexity, such as long documents, an increasing function can be used to increase the distillation weights at lower levels.

[0135] For simple text, higher-level distillation weights are increased, reducing unnecessary computational resource consumption. For complex text, lower-level distillation weights can be increased to ensure the model fully captures the text's details. Setting the weighted coefficients for each distillation layer effectively optimizes computational efficiency, improving the model's inference speed and response time while maintaining accurate duplicate checking.

[0136] Step 203: Input the duplicate check sample pair into the teacher model to obtain a first output result of each distillation level of the teacher model; and input the duplicate check sample pair into the student model to obtain a second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model;

[0137] The embodiment of the present invention can input the duplicate checking sample pairs into two models at the same time, and obtain the first output result of each distillation layer of the teacher model and the second output result of each distillation layer of the student model respectively, so as to facilitate the subsequent compression of the teacher's multi-layer features into the student's single-layer parameters. Through refined hierarchical knowledge transfer, the student model reduces the consumption of computing resources while improving the accuracy of duplicate checking.

[0138] Step 204: determining a correspondence between each distillation level of the teacher model and each distillation level of the student model based on the number of distillation levels of the teacher model and the number of distillation levels of the student model;

[0139] In an embodiment of the present invention, the multi-layer knowledge of the teacher model can be compressed into a single layer of the student model. For example, if the teacher model has L layers and the student model has L' layers, the number of combined layers of the teacher model during the distillation process can be (L / 2) / L'. Assuming (L / 2) / L'=3, the teacher's 2nd to 4th layers can be weighted output to the student's 1st layer, the teacher's 5th to 7th layers can be weighted output to the student's 2nd layer, and the teacher's 8th to 10th layers can be weighted output to the student's 3rd layer. Through the above steps, the correspondence between each distillation level of the teacher model and each distillation level of the student model can be determined.

[0140] Step 205: determining a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model based on a correspondence between each distillation level of the teacher model and each distillation level of the student model;

[0141] In one embodiment, the step of determining the hierarchical distillation loss between the first output result of each distillation level of the teacher model and the second output result of the corresponding distillation level of the student model based on the correspondence between each distillation level of the teacher model and each distillation level of the student model may further include the following sub-steps:

[0142] In sub-step S41, according to the following formula, based on the correspondence between each distillation level of the teacher model and each distillation level of the student model, the hierarchical distillation loss between the first output result of each distillation level of the teacher model and the second output result of the corresponding distillation level of the student model is determined:

[0143] L l (x) = KL(S l′ (x)||T l (x))

[0144] Among them, l is the current distillation level of the teacher model, l′ is the distillation level of the student model corresponding to the current distillation level of the teacher model, and L l (x) is the distillation loss, S l′ (x) is the second output result of the distillation layer of the student model, T l (x) is the first output result of the distillation layer of the teacher model, KL(P||Q)=∑ i P(i)log(P(i) / Q(i)).

[0145] Step 206: performing a weighted summation of the hierarchical distillation losses between the teacher model and the student model according to the weight coefficient of each network layer of the teacher model to obtain the distillation loss between the teacher model and the student model;

[0146] In this embodiment of the present invention, after obtaining the hierarchical distillation loss between each distillation level of the teacher model and the distillation level corresponding to the student model according to the formula in sub-step S41, a weighted summation can be performed according to the weighting coefficient to obtain the total distillation loss between the teacher model and the student model. The formula for weighted summation of the hierarchical distillation loss is as follows:

[0147]

[0148] Among them, Ldistill(x) is the distillation loss between the teacher model and the student model.

[0149] This embodiment of the present invention dynamically adjusts the distillation weights at different levels based on the complexity of the duplicate-checking sample pairs. For simple text, higher-level distillation weights are increased, thereby reducing unnecessary computational resource consumption. For complex text, lower-level distillation weights are increased to ensure the model fully captures the text's details. Setting the weighted coefficients at each distillation level effectively optimizes computational efficiency, improving the model's inference speed and response time while maintaining accurate duplicate-checking accuracy.

[0150] Step 207: determining a parameter gradient of the student model based on the distillation loss between the teacher model and the student model;

[0151] In one embodiment, the step of determining the parameter gradient of the student model based on the distillation loss between the teacher model and the student model may further include the following sub-steps:

[0152] In sub-step S51, the parameter gradient of the student model is determined according to the distillation loss between the teacher model and the student model according to the following formula:

[0153]

[0154] Among them, W is the weight matrix, L is the total network level of the teacher model, l is the current distillation level, α l is the weight coefficient of the current distillation level, L distill is the distillation loss between the teacher model and the student model.

[0155] In an embodiment of the present invention, the gradient direction of parameter update can be calculated through the multi-level distillation loss of the teacher model and the student model. By refining the gradient control, the student model can obtain most of the performance of the teacher model after training.

[0156] Step 208: Obtain a preset learning rate, and update the parameters of the student model according to the preset learning rate and the parameter gradient of the student model;

[0157] In one embodiment, the step of updating the parameters of the student model according to the preset learning rate and the parameter gradient of the student model may further include the following sub-steps:

[0158] Sub-step S61, updating the parameters of the student model according to the preset learning rate and the parameter gradient of the student model according to the following formula:

[0159]

[0160] Among them, θ′ studengt is the parameter of the updated student model, θ student is the parameter of the student model before updating, η is the preset learning rate, is the parameter gradient of the student model.

[0161] In an embodiment of the present invention, the training time of the learning model can be reduced through the synergistic effect of the learning rate and gradient update.

[0162] Step 209, repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as the small language model for duplicate checking.

[0163] In an embodiment of the present invention, the preset condition can be that the distillation loss between the teacher model and the student model has not decreased for three consecutive cycles. Those skilled in the art can set the preset condition to other appropriate conditions based on the idea of ​​the present invention, and the present invention is not limited to this. Through multiple rounds of loss feedback updates, the student model can gradually absorb the deep matching ability of the teacher model, and a small language model for checking for duplicates can be obtained when the student model meets the preset conditions. Since the student model is a small language model and is combined with knowledge distillation technology, the small language model for checking for duplicates can be efficiently run on resource-constrained devices (such as terminal devices). By reducing dependence on server-side computing resources, the small language model for checking for duplicates can effectively reduce hardware resource consumption and reduce operating costs.

[0164] The embodiment of the present invention can obtain duplicate detection sample pairs, and determine the complexity of the duplicate detection sample pairs according to the text features of the duplicate detection sample pairs, thereby overcoming the limitations of traditional duplicate detection methods in evaluating text complexity by analyzing text features; determine the distillation level of the teacher model according to the complexity of the duplicate detection sample pairs, and determine the weighting coefficient of each distillation level of the teacher model, so as to dynamically adjust the allocation of computing resources and improve the distillation efficiency by dynamically associating the text complexity with the knowledge distillation level; input the duplicate detection sample pairs into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate detection sample pairs into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model; according to each level of the teacher model The weighted coefficients of network distillation, the first output results of each distillation layer of the teacher model, and the second output results of each distillation layer of the student model are used to determine the distillation loss between the teacher model and the student model, thereby improving the effect of knowledge distillation in the text duplication detection task through multi-level distillation loss calculation; according to the distillation loss between the teacher model and the student model, the parameters of the student model are updated, thereby optimizing the student model through dynamic distillation and loss feedback; repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as a small language model for duplication detection, so that the student model can gradually absorb the deep matching ability of the teacher model. At the same time, the small language model can avoid excessive consumption of resources and achieve low-power, high-precision duplication detection effects.

[0165] Reference Figure 3 , shows a logic diagram of a small language model training method for checking duplicates combined with multi-level knowledge distillation provided by an embodiment of the present invention, in order to enable those skilled in the art to better understand the embodiment of the present invention, the following is a Figure 3The embodiments of the present invention are described below:

[0166] Step 301, obtaining duplicate checking sample pairs;

[0167] Step 302: Determine the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair;

[0168] Step 303: Determine the weight coefficient of each distillation level of the teacher model according to the complexity of the duplicate check sample pairs;

[0169] Step 304: Input the duplicate check sample pairs into the teacher model to obtain the first output result of each distillation level of the teacher model;

[0170] Step 305: Input the duplicate check sample pair into the student model to obtain the second output result of each distillation level of the student model;

[0171] Step 306 , determining a level-by-level distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model;

[0172] Step 307: Based on the weight coefficient of each network layer of the teacher model, the layer-wise distillation loss between the teacher model and the student model is weighted and summed to obtain the distillation loss between the teacher model and the student model.

[0173] Step 308: updating the parameters of the student model based on the distillation loss between the teacher model and the student model;

[0174] Step 309: If the updated student model meets the preset conditions, the updated student model is used as the small language model for duplicate checking.

[0175] The embodiment of the present invention is implemented through three parts: multi-level distillation, dynamic weighting, and text complexity evaluation. Multi-level distillation distills knowledge from different levels of the teacher model and balances the contribution of each layer through weighting coefficients. Dynamic weighting dynamically adjusts the weighting coefficient of each layer according to the complexity of the input text, so that the model can automatically optimize the distillation process on different types of text. Text complexity evaluation is to evaluate the complexity of the input text, design a reasonable weighting function, and flexibly adjust the depth and strength of the distillation strategy. The embodiment of the present invention can enable the student model to gradually absorb the deep matching capabilities of the teacher model while avoiding excessive consumption of resources, thereby achieving low power consumption and high-precision duplicate checking effects.

[0176] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required for the embodiments of the present invention.

[0177] Reference Figure 4 , shows a structural block diagram of a small language model training device for duplicate checking combined with multi-level knowledge distillation provided by an embodiment of the present invention, which may specifically include the following modules:

[0178] The text complexity determination module 401 is used to obtain a duplicate check sample pair and determine the complexity of the duplicate check sample pair based on the text features of the duplicate check sample pair;

[0179] A text complexity using module 402 is used to determine the distillation level of the teacher model according to the complexity of the duplicate check sample pair, and to determine the weighting coefficient of each distillation level of the teacher model;

[0180] The model result output module 403 is configured to input the duplicate check sample pair into the teacher model to obtain a first output result of each distillation level of the teacher model; and input the duplicate check sample pair into the student model to obtain a second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model;

[0181] a distillation loss determination module 404, configured to determine a distillation loss between the teacher model and the student model based on a weighted coefficient of each network distillation of the teacher model, a first output result of each distillation level of the teacher model, and a second output result of each distillation level of the student model;

[0182] A student model parameter updating module 405 is configured to update the parameters of the student model according to the distillation loss between the teacher model and the student model;

[0183] The duplicate checking language model generation module 406 is used to repeat the above steps until the updated student model meets the preset conditions, and then the updated student model is used as the duplicate checking small language model.

[0184] In an embodiment of the present invention, the text features of the duplicate checking sample pair include text length, lexical diversity, and semantic density. The text complexity determination module 401 includes:

[0185] A preset weight coefficient acquisition submodule is used to obtain preset weight coefficients corresponding to the text length, the lexical diversity and the semantic density;

[0186] A normalized score determination submodule is used to normalize the text length, lexical diversity and semantic density of the duplicate check sample pair respectively to obtain normalized scores of the text length, lexical diversity and semantic density of the duplicate check sample pair;

[0187] The text complexity determination submodule is used to perform weighted summation on the normalized scores of the text length, lexical diversity and semantic density of the duplicate checking sample pair according to the preset weight coefficients corresponding to the text length, lexical diversity and semantic density, so as to obtain the complexity of the duplicate checking sample pair.

[0188] In an embodiment of the present invention, the text complexity using module 402 includes:

[0189] A first distillation level determination submodule is configured to use the first half of the network level of the teacher model as the distillation level of the teacher model when the complexity of the duplicate check sample pair is greater than a preset threshold;

[0190] The second distillation level determination submodule is used to use the second half of the network level of the teacher model as the distillation level of the teacher model when the complexity of the duplicate check sample pair is less than or equal to the preset threshold.

[0191] In the embodiment of the present invention, the text complexity using module 402 further includes:

[0192] The weight coefficient determination submodule is used to determine the weight coefficient of each distillation level of the teacher model according to the following formula:

[0193]

[0194] Among them, l is the current distillation level, fl(C(x)) is the weight coefficient of the current distillation level, L is the total network level of the teacher model, and C(x) is the complexity of the duplicate checking sample pair.

[0195] In this embodiment of the present invention, the distillation loss determination module 404 includes:

[0196] a distillation level correspondence determination submodule, configured to determine a correspondence between each distillation level of the teacher model and each distillation level of the student model based on the number of distillation levels of the teacher model and the number of distillation levels of the student model;

[0197] a hierarchical distillation loss determination submodule, configured to determine, based on a correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model;

[0198] The distillation loss determination submodule is used to perform weighted summation of the hierarchical distillation losses between the teacher model and the student model according to the weight coefficient of each network layer of the teacher model to obtain the distillation loss between the teacher model and the student model.

[0199] In this embodiment of the present invention, the hierarchical distillation loss determination submodule includes:

[0200] A hierarchical distillation loss determining unit is configured to determine, according to the following formula, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model based on a correspondence between each distillation level of the teacher model and each distillation level of the student model:

[0201] L l (x) = KL(S l′ (x)||T l (x))

[0202] Among them, l is the current distillation level of the teacher model, l′ is the distillation level of the student model corresponding to the current distillation level of the teacher model, and L l (x) is the distillation loss, S l′ (x) is the second output result of the distillation layer of the student model, T l (x) is the first output result of the distillation layer of the teacher model, KL(P||Q)=∑ i P(i)log(P(i) / Q(i)).

[0203] In this embodiment of the present invention, the student model parameter updating module 405 includes:

[0204] The preset learning rate acquisition submodule is used to obtain the preset learning rate;

[0205] a parameter gradient determination submodule, configured to determine a parameter gradient of the student model based on the distillation loss between the teacher model and the student model;

[0206] The student model parameter updating submodule is used to update the parameters of the student model according to the preset learning rate and the parameter gradient of the student model.

[0207] In an embodiment of the present invention, the parameter gradient determination submodule includes:

[0208] A parameter gradient determination unit is configured to determine the parameter gradient of the student model according to the distillation loss between the teacher model and the student model according to the following formula:

[0209]

[0210] Among them, W is the weight matrix, L is the total network level of the teacher model, l is the current distillation level, α l is the weight coefficient of the current distillation level, L distill is the distillation loss between the teacher model and the student model;

[0211] The student model parameter updating submodule includes:

[0212] The student model parameter updating unit is used to update the parameters of the student model according to the preset learning rate and the parameter gradient of the student model according to the following formula:

[0213]

[0214] Among them, θ′ studengt is the parameter of the updated student model, θ student is the parameter of the student model before updating, η is the preset learning rate, is the parameter gradient of the student model.

[0215] In this embodiment of the present invention, the duplicate checking language model generation module 406 includes:

[0216] A preset condition judgment submodule is used to repeat the above steps until the distillation loss between the teacher model and the student model does not decrease within a preset distillation period, and then determine that the updated student model meets the preset condition;

[0217] The duplicate checking language model generation submodule is used to use the updated student model as a duplicate checking small language model.

[0218] A small language model training device for checking for duplicate content combined with multi-level knowledge distillation in an embodiment of the present invention can obtain duplicate sample pairs and determine the complexity of the duplicate sample pairs based on the text features of the duplicate sample pairs, thereby overcoming the limitations of traditional duplicate checking methods in evaluating text complexity by analyzing text features; determine the distillation level of the teacher model based on the complexity of the duplicate sample pairs, and determine the weighting coefficient of each distillation level of the teacher model, so as to dynamically adjust the allocation of computing resources and improve the distillation efficiency by dynamically associating the text complexity with the knowledge distillation level; input the duplicate sample pairs into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate sample pairs into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model. language model; determine the distillation loss between the teacher model and the student model according to the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation layer of the teacher model, and the second output result of each distillation layer of the student model, thereby improving the effect of knowledge distillation in the text duplication detection task through multi-level distillation loss calculation; update the parameters of the student model according to the distillation loss between the teacher model and the student model, thereby optimizing the student model through dynamic distillation and loss feedback; repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as a small language model for duplication detection, so that the student model gradually absorbs the deep matching ability of the teacher model. At the same time, the small language model can avoid excessive consumption of resources and achieve low-power, high-precision duplication detection effect.

[0219] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0220] An embodiment of the present invention also provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for training a small language model for duplicate detection combined with multi-level knowledge distillation are implemented.

[0221] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of the above-mentioned method for training a small language model for duplicate detection combined with multi-level knowledge distillation.

[0222] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0223] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more machine-readable media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0224] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems) and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0225] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0226] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0227] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0228] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0229] The above is a detailed introduction to a small language model training method for checking for duplicate content combined with multi-level knowledge distillation and a small language model training device for checking for duplicate content combined with multi-level knowledge distillation provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for training a small language model for duplicate checking combined with multi-level knowledge distillation, characterized in that: The method comprises: Obtaining a duplicate checking sample pair, and determining the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair; Determining a distillation level of a teacher model according to the complexity of the duplicate checking sample pair, and determining a weighting coefficient of each distillation level of the teacher model; Input the duplicate checking sample pair into the teacher model to obtain the first output result of each distillation level of the teacher model; and input the duplicate checking sample pair into the student model to obtain the second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model; Determining a distillation loss between the teacher model and the student model according to a weighted coefficient of each network distillation of the teacher model, a first output result of each distillation level of the teacher model, and a second output result of each distillation level of the student model; updating the parameters of the student model according to the distillation loss between the teacher model and the student model; Repeat the above steps until the updated student model meets the preset conditions, and then use the updated student model as the small language model for checking for duplicates.

2. The method according to claim 1, characterized in that The text features of the duplicate checking sample pair include text length, lexical diversity, and semantic density. Determining the complexity of the duplicate checking sample pair based on the text features of the duplicate checking sample pair includes: Obtaining preset weight coefficients corresponding to the text length, the lexical diversity, and the semantic density; Normalizing the text length, lexical diversity, and semantic density of the duplicate checking sample pairs respectively to obtain normalized scores of the text length, lexical diversity, and semantic density of the duplicate checking sample pairs; According to the preset weight coefficients corresponding to the text length, the lexical diversity and the semantic density, the normalized scores of the text length, lexical diversity and semantic density of the duplicate checking sample pair are weighted and summed to obtain the complexity of the duplicate checking sample pair.

3. The method according to claim 1, characterized in that Determining the distillation level of the teacher model according to the complexity of the duplicate checking sample pair includes: When the complexity of the duplicate checking sample pair is greater than a preset threshold, the first half of the network layer of the teacher model is used as the distillation layer of the teacher model; When the complexity of the duplicate check sample pair is less than or equal to the preset threshold, the second half of the network layer of the teacher model is used as the distillation layer of the teacher model.

4. The method according to claim 1, wherein Determining the weight coefficient of each distillation level of the teacher model includes: The weight coefficient of each distillation level of the teacher model is determined according to the following formula: Among them, l is the current distillation level, fl(C(x)) is the weight coefficient of the current distillation level, L is the total network level of the teacher model, and C(x) is the complexity of the duplicate checking sample pair.

5. The method according to claim 1, characterized in that Determining the distillation loss between the teacher model and the student model according to the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model includes: Determining a correspondence between each distillation level of the teacher model and each distillation level of the student model according to the number of distillation levels of the teacher model and the number of distillation levels of the student model; Determining, according to a correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of the corresponding distillation level of the student model; According to the weighted coefficient of each network layer of the teacher model, the hierarchical distillation loss between the teacher model and the student model is weightedly summed to obtain the distillation loss between the teacher model and the student model.

6. The method according to claim 5, characterized in that The determining, according to the correspondence between each distillation level of the teacher model and each distillation level of the student model, a hierarchical distillation loss between a first output result of each distillation level of the teacher model and a second output result of a corresponding distillation level of the student model, includes: According to the following formula, based on the corresponding relationship between each distillation level of the teacher model and each distillation level of the student model, the hierarchical distillation loss between the first output result of each distillation level of the teacher model and the second output result of the corresponding distillation level of the student model is determined: L l (x)=KL(S l′ (x)||T l (x)) Among them, l is the current distillation level of the teacher model, l′ is the distillation level of the student model corresponding to the current distillation level of the teacher model, and L l (x) is the distillation loss, S l′ (x) is the second output result of the distillation layer of the student model, T l (x) is the first output result of the distillation layer of the teacher model, KL(P||Q)=∑ i P(i)log(P(i) / Q(i)).

7. The method according to claim 1, characterized in that The updating of the parameters of the student model according to the distillation loss between the teacher model and the student model includes: Get the preset learning rate; Determining a parameter gradient of the student model according to the distillation loss between the teacher model and the student model; The parameters of the student model are updated according to the preset learning rate and the parameter gradient of the student model.

8. The method according to claim 7, characterized in that Determining the parameter gradient of the student model according to the distillation loss between the teacher model and the student model includes: The parameter gradient of the student model is determined according to the distillation loss between the teacher model and the student model according to the following formula: Among them, W is the weight matrix, L is the total network level of the teacher model, l is the current distillation level, α l is the weight coefficient of the current distillation level, L distill is the distillation loss between the teacher model and the student model; The updating of the parameters of the student model according to the preset learning rate and the parameter gradient of the student model includes: According to the following formula, the parameters of the student model are updated according to the preset learning rate and the parameter gradient of the student model: Among them, θ′ studengt is the parameter of the updated student model, θ student is the parameter of the student model before updating, η is the preset learning rate, is the parameter gradient of the student model.

9. The method according to claim 1, characterized in that Repeating the above steps until the updated student model meets the preset conditions, and then using the updated student model as the small language model for duplicate checking, including: Repeat the above steps until the distillation loss between the teacher model and the student model does not decrease within a preset distillation period, then determine that the updated student model meets the preset condition; The updated student model is used as a small language model for checking for duplicate content.

10. A small language model training device for checking duplicate content combined with multi-level knowledge distillation, characterized in that: The device comprises: A text complexity determination module is used to obtain a duplicate check sample pair and determine the complexity of the duplicate check sample pair based on the text features of the duplicate check sample pair; A text complexity utilization module, configured to determine a distillation level of a teacher model according to the complexity of the duplicate check sample pair, and to determine a weighting coefficient of each distillation level of the teacher model; A model result output module is configured to input the duplicate check sample pair into the teacher model to obtain a first output result of each distillation level of the teacher model; and input the duplicate check sample pair into the student model to obtain a second output result of each distillation level of the student model; the teacher model is a large language model, and the student model is a small language model; a distillation loss determination module, configured to determine the distillation loss between the teacher model and the student model based on the weighted coefficient of each network distillation of the teacher model, the first output result of each distillation level of the teacher model, and the second output result of each distillation level of the student model; A student model parameter updating module, configured to update the parameters of the student model according to the distillation loss between the teacher model and the student model; The duplicate checking language model generation module is used to repeat the above steps until the updated student model meets the preset conditions, and then the updated student model is used as the duplicate checking small language model.

11. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of a small language model training method for duplicate detection combined with multi-level knowledge distillation are implemented as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, it implements the steps of a small language model training method for duplicate detection combined with multi-level knowledge distillation as described in any one of claims 1 to 9.

Images