Natural language interactive learning method for large language model in digital commemorative inspection industry
By grading and word-partitioning statistical weight processing, the distributed feature vector is constructed and TF-IDF weights are replaced, the problem of high error rate of Simhash algorithm in short text plagiarism checking is solved, and the interaction learning efficiency of large language models is improved.
Patent Information
- Application Number
- CN202411006892.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-29
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional Simhash algorithm has poor dilution checking effect when processing short texts, resulting in low interactive learning efficiency of large language models in the digital discipline inspection industry.
The text is graded, and the distribution feature vector is constructed based on the text hierarchical weights and word segmentation similarity statistical weights. The distribution divergence of participle segmentation is replaced by the Simhash code calculation is improved, and the phenomenon of short text features ignore and pay attention to long text features.
It improves the accuracy of short text plagiarism checking, reduces the repetition rate of corpus, and improves the learning efficiency of large language models.
Smart Images

Figure CN119046403A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of natural language processing technology, and specifically to a natural language interactive learning method for a large language model in the digital discipline inspection industry. Background Art
[0002] The digital discipline inspection industry needs to process a large amount of text data. Manual processing is difficult and prone to errors. Therefore, it can be assisted by a large language model to improve the efficiency and accuracy of text processing. When assisting staff to process digital discipline inspection text data, the assistant will give the interactive text to the large language model for reading. In order to reduce the amount of text processing by the large language model, it is necessary to check the interactive text for duplicates in advance to avoid repeated processing of a large number of similar texts, which reduces the efficiency of natural language interactive learning.
[0003] The Simhash algorithm is a common similar text processing algorithm in the field of natural language processing. The traditional Simhash algorithm uses TF-IDF weights to weight different word segments in the text. When the text is short, the local sensitive hash value of the text may not be able to distinguish the text well due to the lack of highly discriminative word segments in the text, which in turn leads to errors in text duplication detection. Therefore, when processing natural language interactive learning text in the digital discipline inspection industry, there is a problem that the Simhash algorithm has a poor duplication detection effect due to the shortness of the text, resulting in errors in text duplication detection and reducing the efficiency of interactive learning of large language models. Summary of the invention
[0004] In order to solve the above technical problems, this application provides a natural language interactive learning method for a large language model in the digital discipline inspection industry to solve existing problems.
[0005] The natural language interactive learning method for the large language model of the digital discipline inspection industry in this application adopts the following technical solutions:
[0006] An embodiment of the present application provides a natural language interactive learning method for a large language model in the digital discipline inspection industry, the method comprising the following steps:
[0007] Obtain natural language interactive text data of the digital discipline inspection industry and corpus text data in the large prediction model corpus, and perform word segmentation on the text data;
[0008] All texts are graded according to the number of words in the texts, and the text grade weight of each text grade is determined based on the number of occurrences of each word in each text at each text grade;
[0009] Determine the similarity statistical weight between each segmentation word and each similar segmentation word at each text level by combining the text level weight of each text level and the difference in the number of characters between each segmentation word and each similar segmentation word at each text level;
[0010] Each text at each text level is evenly divided into several segments according to the number of words; based on the number of occurrences of each segmentation word and all similar segmentations in each segment in all texts at each text level and the similarity statistical weights between them, a distribution feature vector of each segmentation word at each text level is constructed;
[0011] Determine the distribution divergence of each segmentation in the interactive text by combining the text level weight differences between the text levels and the differences between the distribution feature vectors of each segmentation in the interactive text and the corresponding segmentation in all text levels;
[0012] The Simhash code generated by each corpus text and interactive text in the corpus is improved based on distribution divergence, and the large language model of the data discipline inspection industry is trained based on the similarity between the improved Simhash codes.
[0013] Preferably, the method for determining the text level weight of each text level includes:
[0014] For each text at each text level, count the number of occurrences of each word segmentation in each text;
[0015] Count the number of occurrences of each type in each text as the value of the element corresponding to the position in the arrangement result vector; the corresponding position is the value corresponding to the number of occurrences of each type;
[0016] The text level weight of each text level is determined based on the fractal dimension of the arrangement result vector of all texts under each text level.
[0017] Preferably, the text level weight of each text level is determined based on the fractal dimension of the arrangement result vector of all texts at each text level, and the expression is: Among them, Tl n is the text level weight of the nth text level, M is the number of texts in the nth text level, and Fd m,n It is the fractal dimension of the permutation result vector of the mth text at the nth text level.
[0018] Preferably, the method for determining the similarity statistical weight between each segmentation and each similar segmentation at each text level comprises:
[0019] For each word at each text level, the words with the same characters are taken as similar words of each word;
[0020] Determine the character similarity between each segmentation word and each similar segmentation word based on the difference in the number of characters between each segmentation word and each similar segmentation word;
[0021] The similarity statistical weight between each participle and each similar participle at each text level is determined by combining the text level weight of each text level and the character similarity between each participle and each similar participle at each text level.
[0022] Preferably, the expression of the character similarity between each participle and each similar participle is: In the formula, sm s,t is the character similarity between the s-th participle and its t-th similar participle, Sn s,t is the number of characters that are common between the s-th participle and its t-th similar participle, Cn t is the number of characters of the tth similar participle of the sth participle, Cn s is the number of characters in the sth word segmentation, min is the minimum function, and max is the maximum function.
[0023] Preferably, the combining of the text level weight of each text level and the character similarity between each segmentation and each similar segmentation at each text level to determine the similarity statistical weight between each segmentation and each similar segmentation at each text level includes:
[0024] Calculate the difference between the number 1 and the text level weight for each text level;
[0025] The product of the character similarity between each segmentation and each similar segmentation at each text level and the difference is used as the similarity statistical weight between each segmentation and each similar segmentation at each text level.
[0026] Preferably, the process of constructing the distribution feature vector of each word segmentation at each text level includes:
[0027] Determine the number of repetitions of each participle in each paragraph by combining the number of occurrences of each participle in each paragraph with all its similar participles and the similarity statistical weights between them;
[0028] Constructing a distribution statistics vector for each word segment of each text; the length of the distribution statistics vector is the same as the number of segment segments;
[0029] The number of repetitions of each word segment in each paragraph in each text is used as the element value of the position of the corresponding paragraph in the distribution statistics vector;
[0030] The mean vector of the normalized distribution statistical vectors of each word segmentation of all texts at each text level is used as the distribution feature vector of each word segmentation at each text level.
[0031] Preferably, the method for determining the number of repetitions of each word segment in each segment includes:
[0032] Counting the number of occurrences of each participle in each paragraph; counting the number of occurrences of each similar participle to each participle in each paragraph;
[0033] Calculate the number of occurrences of each similar participle of each participle in each paragraph and the product of the similarity statistical weight between each participle and each similar participle, and record it as the first product;
[0034] Calculating the cumulative sum of the first products of all similar participles of each participle in each segment;
[0035] The sum of the accumulated sum and the number of occurrences of each word segment in each paragraph is used as the number of repetitions of each word segment in each paragraph.
[0036] Preferably, the expression of the distribution divergence of each word segmentation in the interactive text is: Among them, FS s is the distribution divergence of the s-th word segmentation in the interactive text, It is the operation of rounding up the number in the brackets. N is the maximum text level. DS n,s is the difference between the distribution feature vector of the sth word segmentation in the interactive text and the corresponding word segmentation in the nth text level in the corpus, norm is the normalization function, Tl n is the text level weight of the nth text level, Tl h is the text level weight of the text level where the interactive text is located, and μ is the preset correction parameter.
[0037] Preferably, the Simhash code generated by each corpus text and interactive text in the corpus is improved based on the distribution divergence, and the large language model of the discipline inspection industry is trained based on the similarity between the improved Simhash codes, including:
[0038] The distribution divergence of each word segmentation replaces the TF-IDF weight in the Simhash algorithm, and a corresponding improved Simhash code is generated for each corpus text and interactive text in the corpus;
[0039] The corpus is adjusted according to the similarity between each corpus text and the improved Simhash code of the interactive text, and the adjusted corpus is used as the input of the large language model for training, and the updated digital discipline inspection industry large language model is output;
[0040] When the similarity is less than a preset threshold, the interactive text is added to the corpus; otherwise, no processing is performed.
[0041] This application has at least the following beneficial effects:
[0042] In view of the high error rate of short text duplicate checking faced by the traditional Simhash algorithm and the reduced interactive learning efficiency of the large language model, the present application proposes a natural language interactive learning method for the large language model in the digital discipline inspection industry, including: grading texts of different lengths, and calculating text grade weights according to the statistical features of the texts to characterize the length differences of the texts; calculating similar statistical weights for similar word segmentations in combination with the text grade features, and when one of two similar word segmentations is repeated itself, the other word segmentation is also counted as being repeated, thereby avoiding statistical errors caused by the small number of characters in the short text; finally, in combination with the similar statistical weights, a distribution feature vector is calculated by the position distribution of the word segmentations in the text, and the word segmentation distribution divergence is calculated by the difference between the distribution feature vector of the text and the distribution feature vectors of different text grades, thereby replacing the TF-IDF weight adopted by the traditional Simhash algorithm, thereby avoiding the situation in which the short text features are ignored and the long text features are emphasized when extracting text-related word segmentation features due to the different text lengths. Finally, the word distribution divergence is used instead of the TF-IDF weight as the calculation weight of Simhash to complete the duplicate checking of the interactive text, reduce the text repetition rate of the corpus, and improve the learning efficiency of the large language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0044] Figure 1 A flowchart of a natural language interactive learning method for a large language model in the digital discipline inspection industry provided by an embodiment of the present application;
[0045] Figure 2 A flow chart of a method for determining a text level weight for each text level provided in one embodiment of the present application;
[0046] Figure 3 A flow chart of a method for determining similarity statistical weights between each segmentation and each similar segmentation at each text level provided by an embodiment of the present application;
[0047] Figure 4 A schematic diagram of a specific improved method for generating an improved Simhash feature code provided in one embodiment of the present application. DETAILED DESCRIPTION
[0048] In order to further explain the technical means and effects adopted by this application to achieve the predetermined invention purpose, the natural language interactive learning method for the large language model of the digital discipline inspection industry proposed in this application, its specific implementation method, structure, characteristics and effects are described in detail as follows in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.
[0049] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
[0050] The specific scheme of the natural language interactive learning method for the large language model of the digital discipline inspection industry provided by this application is described in detail below with reference to the accompanying drawings.
[0051] An embodiment of the present application provides a natural language interactive learning method for a large language model in the digital discipline inspection industry.
[0052] Specifically, the following natural language interactive learning method for the large language model in the digital discipline inspection industry is provided. Please refer to Figure 1 , the method comprises the following steps:
[0053] The first step is to obtain the natural language interaction text data of the digital discipline inspection industry and the corpus text data in the large prediction model corpus, and perform word segmentation on the text data.
[0054] This embodiment records the natural language interactive text data given by the discipline inspection personnel, and compares the recorded text data with the text data in the corpus for similarity. The similarity comparison steps include:
[0055] The traditional Simhash algorithm generates a local sensitive hash value for the interactive text, and quickly compares the local sensitive hash value of the interactive text with the local sensitive hash values of all texts in the corpus to find similar texts. The similar texts are further compared with the interactive texts to check for duplicates. If they are repeated, no processing is done; if they are not repeated, the interactive text is added to the corpus.
[0056] Therefore, this embodiment obtains all text data in the large language model corpus through the large language model corpus, wherein one piece of text data is recorded as one corpus text, and at the same time obtains the interactive text given by the user of the large language model, which is recorded as one interactive text.
[0057] The large language model of this embodiment uses Chinese text data. The first step in processing Chinese text is word segmentation. Therefore, in this embodiment, interactive text and corpus text are used as inputs, and the bidirectional maximum matching algorithm is used for processing. The output is the interactive text and corpus text after word segmentation. There are multiple identifiers in the output text, and there is a single word between two identifiers. Among them, the bidirectional maximum matching algorithm is a commonly used technology in the field of natural language processing, and will not be elaborated in this embodiment.
[0058] In the second step, all texts are classified according to the word count differences in the texts, and the text level weights of each text level are determined based on the occurrence times of word segmentation within each text under each level.
[0059] This embodiment solves the problem of inaccurate duplicate checking when the Simhash algorithm faces short text duplicate checking. The main reason for the poor effect of the Simhash algorithm in processing short texts is the small amount of short text data. The traditional Simhash algorithm uses TF-IDF weights when assigning weights to word segmentation. TF-IDF weights are assigned based on the data of the entire text, which results in TF-IDF weights being more inclined to represent the characteristic words of long texts, and thus leads to poor processing ability of the Simhash algorithm for short texts. Among them, TF-IDF weights are a commonly used technology in the field of natural language processing, and will not be elaborated in this embodiment.
[0060] In this embodiment, the texts are classified according to the word count of the texts. At the same time, the word segmentation repetition feature of the text itself is used as the feature value of different text levels. The better the word segmentation repetition feature of a text level, it indicates that the word segmentation statistical law of this text level is more in line with the law of large numbers, and objectively, the text of this text level belongs to a long text; and the greater the difference in repetition features between two text levels, it means that when using the same feature words for duplicate checking between the two levels, the local sensitive features of the shorter text are more likely to be ignored.
[0061] Through text classification processing, long and short texts are distinguished, different word segmentation weights are assigned, and the text duplicate checking errors caused by different text lengths are reduced.
[0062] In this embodiment, the texts are classified according to the word count of the texts. Optionally, the classification method in this embodiment is specifically as follows:
[0063] When the word count of the text is 500 or less, the text is classified as a text of the first level;
[0064] When 2 ≤ n < N, that is, when the word count of the text is between 500×2 n-2 and 500×2 n-1 the text is classified as a text of the nth level;
[0065] When the word count of the text is greater than 500×2 N-1, divide the text into N-th level text.
[0066] In this embodiment, the maximum text level is set to N=10, and the implementer can set the maximum text level according to actual conditions.
[0067] In this embodiment, the text level weight of each text level is calculated according to the number of occurrences of the segmentation words in each text level. Specifically:
[0068] Furthermore, we obtain all the texts at the nth text level in the corpus. Assuming that the mth text at the nth text level has S kinds of participles, we count the number of occurrences of the sth participle in the mth text and record it as Num. s .
[0069] For each text at each text level, the number of occurrences of each word segment in each text is counted; the number of each occurrence in each text is counted as the element value of the corresponding position in the arrangement result vector; the corresponding position is the value corresponding to each occurrence.
[0070] In this embodiment, specifically, the number of occurrences of the S types of word segments in the mth text is statistically arranged. The specific method of statistical arrangement is: generate a vector of length R with all elements being 0, recorded as an arrangement record vector, traverse the number of occurrences of the S types of word segments in the mth text, assuming that the number of occurrences of the sth type of word segment is Num s , then add 1 to the rth element of the permutation record vector; after traversing, the permutation result vector is obtained.
[0071] For example, there are 14 kinds of word segmentations in a text, and their appearance times are 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 8, 8, 9, 10 respectively. Then the corresponding arrangement result vector is (0, 1, 3, 3, 3, 0, 0, 2, 1, 1), where the element position r represents the appearance times of the word segmentation, and the element value represents the number of word segmentations with the same appearance times r.
[0072] Finally, an arrangement result vector of length R is obtained. The smoother the data changes of the vector, the more statistically the vector conforms to the law of large numbers, and the corresponding m-th text is more likely to be the longest text.
[0073] Therefore, the result vector of the arrangement of the mth text is used as input, and the Higuchi algorithm is used to calculate, and the output is the fractal dimension Fd of the mth arrangement result vector. m,n , the larger the fractal dimension, the less smooth the record vector of the mth text, the less statistically consistent the mth text is with the law of large numbers, and the shorter the text is. Among them, the Higuchi algorithm is a well-known technology and will not be described in detail.
[0074] Based on the fractal dimension of the arrangement result vector of all texts at each text level, the text level weight of each text level is determined, and the expression is: n =∑ M m=1 (Fd m,n -1); where Tl n is the text level weight of the nth text level, M is the number of texts in the nth text level, and Fd m,n It is the fractal dimension of the permutation result vector of the mth text at the nth text level.
[0075] Among them, since the arrangement result vector is a one-dimensional vector, its fractal dimension is between 1 and 2, and subtracting 1 from it controls its value within the range of 0 to 1. The greater the difference in text level weights between two different text levels, the greater the difference in the number of text words in the statistical sense between the two text levels, and the more different word segmentation weights should be used when the two texts are matched using the Simhash algorithm to avoid incorrect expression of local sensitive features of short texts.
[0076] In one embodiment of the present application, a flow chart of a method for determining a text level weight for each text level is shown in the attached figure. Figure 2 shown.
[0077] The third step is to determine the similarity statistical weight between each segmentation and its similar segmentation at each text level by combining the text level weight of each text level and the difference in the number of characters between each segmentation and its similar segmentation at each text level;
[0078] For text levels with smaller text level weights, the word segmentation repeatability is poor in a statistical sense, resulting in a large error in the extraction of local sensitive features of the text when the TF-IDF weight is used as the weight. Therefore, this embodiment changes the repeated statistical method of word segmentation. For the sth word segmentation, other similar word segmentations are also counted as repetitions of the sth word segmentation, thereby reducing the problem of large errors in the extraction of local sensitive features of short texts caused by insufficient statistical quantity.
[0079] When other participles are counted as repetitions of the sth participle, the two participles are compared based on their similarity. The more similar the two participles are, the more they should be counted as repetitions of the other participle when they repeat themselves.
[0080] Therefore, for the s-th word, we obtain all T-th words that have the same characters as it, and record them as similar words to the s-th word. Assume that the number of characters of the t-th similar word to the s-th word is Cn t , the number of characters in the sth word segmentation is Cn s , the number of characters in common between the s-th participle and its t-th similar participle is Sn s,t, calculate the character similarity between the t-th participle and its s-th similar participle, the expression is: In the formula, sm s,t is the character similarity between the s-th participle and its t-th similar participle, Sn s,t is the number of characters that are common between the s-th participle and its t-th similar participle, Cn t is the number of characters of the tth similar participle of the sth participle, Cn s is the number of characters in the sth word segmentation, min is the minimum function, and max is the maximum function.
[0081] Among them, the greater the proportion of the number of identical characters to the sum of the number of characters of the two participles, the more similar the two participles are, and the higher the character similarity; the closer the ratio of the number of characters of the two participles is to 1, the closer the two participles are in character length, and the higher the character similarity.
[0082] Furthermore, for texts with smaller text level weights, the more serious the statistical error of local sensitive features of the text caused by the length of the text, the more similar segmented words should be counted, and the corresponding similarity statistical weight should be larger. Therefore, combining the text level weight of each text level and the character similarity between each segmented word and each similar segmented word at each text level, the similarity statistical weight between each segmented word and each similar segmented word at each text level is determined. The specific expression is: β s,t,n =sm n,s,t ×(1-Tl n ), where β s,t,n is the similarity statistical weight between the s-th segmentation and its t-th similar segmentation at the n-th text level, sm n,s,t is the character similarity between the sth word segmentation and its tth similar word segmentation at the nth text level, Tl n is the text level weight of the nth text level.
[0083] It should be understood that the greater the character similarity, the more similar the two word segmentations are, and the greater the similarity statistical weight should be; the smaller the text level weight, the shorter the text, the easier it is to produce statistical errors due to insufficient samples, and the greater the similarity statistical weight should be; the greater the similarity statistical weight, the more when two word segmentations repeat themselves, they should be regarded as another word segmentation being repeated.
[0084] In one embodiment of the present application, a flow chart of a method for determining similarity statistical weights between each segmentation and each similar segmentation at each text level is shown in the attached figure. Figure 3 shown.
[0085] The fourth step is to evenly divide each text at each text level into several segments according to the number of words; based on the number of occurrences of each segmentation and all its similar segmentations in each segment in all texts at each text level and the similarity statistical weights between them, a distribution feature vector of each segmentation at each text level is constructed.
[0086] The traditional Simhash algorithm uses TF-IDF weight as the word segmentation weight. TF-IDF only considers the repetition characteristics of the word segmentation, and believes that the fewer the number of repetitions of the word segmentation, the more the word segmentation can distinguish the feature words of different texts, but does not consider the position differences of the word segmentation in different texts.
[0087] For example, if a segmentation word is at the front end of the text in most texts, but at the back end of the text in this text, then in this text, the segmentation word can also be regarded as a text feature word.
[0088] When faced with short texts, it is more difficult to count the difference in the number of segmented words due to the small amount of text. Therefore, this embodiment uses the distribution position of the segmented words in the text instead of the quantitative feature of the segmented words as the weight feature of the segmented words for extraction; since the segmented word distribution feature is a normalized feature, it has nothing to do with the total number of segmented words in the text. It can avoid the situation where the frequency of short text feature words is low and the frequency of long text feature words is high due to the different lengths of the text, and the short text features are ignored while the long text features are emphasized. This helps to solve the problem of large duplicate checking errors in the traditional Simhash algorithm when facing short text duplicate checking.
[0089] In this embodiment, the construction process of the distribution feature vector of each word segmentation at each text level is specifically as follows:
[0090] First, we obtain the distribution statistics vector for a word segmentation in a text by:
[0091] For the s-th word segmentation of the m-th text at the n-th text level, a vector with a length of K is generated and recorded as a distribution statistical vector. In this embodiment, K is 10.
[0092] Divide the mth text into K segments evenly according to the number of words, and count the number of the sth type of segmentation in the kth segment of the mth text at the nth text level, denoted as NC s,k , count the number of similar segmented words in the tth segmented word of the sth segmented word in the kth paragraph of the mth text at the nth text level, and record it as NC t,k , combined with the number of occurrences of each participle in each paragraph and all its similar participles and the similarity statistical weights between them, determine the number of repetitions of each participle in each paragraph. The specific expression is:
[0093]
[0094] Among them, Rnn,m,s,k is the number of repetitions of the sth word in the kth paragraph of the mth text at the nth text level, β s,t,n is the similarity statistical weight between the s-th segmentation and its t-th similar segmentation at the n-th text level; NC s,k It is the number of the sth type of segmentation in the kth paragraph of the mth text at the nth text level.
[0095] It should be understood that, in the kth paragraph of the mth text at the nth text level, the more the sth participle appears, and the more its similar participles appear, and the larger the similarity statistical weight between it and its similar participles is, the more similar they are, and the larger the weight of the number of similar participles appearing, which means that in the kth paragraph of the mth text at the nth text level, the more the sth participle and its similar participles appear, that is, the more repetitions of the sth participle in the kth paragraph of the mth text at the nth text level are.
[0096] Furthermore, the kth element position in the distribution statistics vector is set to the number of repetitions of the sth word segment in the kth paragraph, and the above operation is repeated to fill each element position in the distribution statistics vector according to the number of repetitions of the sth word segment in the corresponding paragraph.
[0097] Then the distribution statistical vector is normalized to obtain the distribution feature vector of the s-th word segmentation in the m-th text.
[0098] Furthermore, the distribution feature vector of the s-th word segmentation is calculated for all texts at the n-th text level in the corpus, and the mean vector of all the distribution feature vectors of the s-th word segmentation at the n-th text level is calculated as the distribution feature vector of the s-th word segmentation at the n-th text level.
[0099] In the fifth step, the distribution divergence of each word segmentation in the interactive text is determined by combining the differences in text-level weights between text levels and the differences in the distribution feature vectors between each word segmentation in the interactive text and the corresponding word segmentation in all text levels.
[0100] Finally, when checking for duplicate content of an interactive text, first obtain its text level h, and at the same time obtain the distribution feature vector of the sth word segmentation at the text level where the interactive text is located. The difference between the distribution feature vector of the sth word segmentation at the text level where the interactive text is located and the distribution feature vector of the corresponding word segmentation at the nth text level is recorded as the word segmentation distribution difference DS between the sth word segmentation in the interactive text and the corresponding word segmentation at the nth text level in the corpus n,s , DS n,sThe larger the value, the more obvious the difference in position distribution between the interactive text and the text of the nth text level in the corpus on the sth word segmentation. When calculating the difference, the JS divergence is used to calculate the difference between the two distribution feature vectors. The calculation of the JS divergence is a common technology in the field of machine learning and will not be repeated in this embodiment.
[0101] In this embodiment, the specific calculation expression of the distribution divergence of each word segmentation in the interactive text is:
[0102]
[0103] Among them, FS s is the distribution divergence of the s-th word segmentation in the interactive text, It is the operation of rounding up the number in the brackets. N is the maximum text level. DS n,s is the difference between the distribution feature vector of the sth word segmentation in the interactive text and the corresponding word segmentation in the nth text level in the corpus, norm is the normalization function, Tl n is the text level weight of the nth text level, Tl h is the text level weight of the text level where the interactive text is located, μ is a preset correction parameter to prevent the denominator from being 0. In this embodiment, the preset correction parameter μ is set to 1, and the implementer can set it according to the actual situation.
[0104] It should be understood that the greater the difference in text-level feature weights between the text-level of the interactive text and the text of the nth text-level in the corpus, the more obvious the length difference between the two texts in statistical features, and the smaller the weight of the distribution feature vector of the nth text-level when judging the characteristic words of the interactive text; the greater the difference in word distribution, the more the sth word in the interactive text is the characteristic text of the interactive text, and the greater the word distribution divergence. The greater the word distribution divergence, the more the sth word is the characteristic word of the interactive text, and the greater the weight of the sth word in the interactive text when checking for duplicates using the Simhash algorithm.
[0105] The sixth step is to improve the Simhash code generated by each corpus text and interactive text in the corpus based on the distribution divergence, and train the large language model of the data discipline inspection industry based on the similarity between the improved Simhash codes.
[0106] This embodiment uses the distribution divergence of word segments to replace the TF-IDF weight in the traditional Simhash algorithm to improve the Simhash algorithm's effect on short text duplication detection.
[0107] The specific improvement method for generating an improved Simhash signature for a text is shown in the attached Figure 4 shown.
[0108] exist Figure 4The specific steps of the improvement method are as follows:
[0109] Step 1: Generate a Simhash signature code from a text, first generate a vector of length Q with all elements being 0, recorded as a hash record vector. In this embodiment, the hash code length Q=64.
[0110] Step 2: Calculate the distribution divergence for each word in the target text, where the distribution divergence of the sth word is recorded as FS s .
[0111] Step 3: Traverse the target text in units of word segments. When the traversed text is the sth word segment, use the sth word segment as input and use the hash value generation algorithm to generate a hash value of length Q. The generated hash value is a binary number. Replace the 0 numbers in the generated hash value with -1, and then multiply each number in the hash value by the distribution divergence FS. s , recorded as a hash code; finally, the hash record vector is added with the hash code to complete the record of the word segmentation. The hash value generation algorithm is a common technology in the field of data processing.
[0112] Step 4: After traversing the text, obtain the final hash record vector, set the negative value position to 0, and the positive value position to 1, and record it as the improved Simhash code of the target text.
[0113] An improved Simhash code is generated for each corpus text in the corpus, and an improved Simhash code is generated for the interactive text. The Simhash of the interactive text is compared with the Simhash of the corpus text. In this embodiment, when the Simhash codes of the two texts have W bits in common, the two texts are considered similar. In this embodiment, W is 50.
[0114] The interactive texts submitted by the interactive personnel are checked for duplication. If there is a corpus text similar to the interactive text, the interactive text submitted by the interactive personnel will not be processed. Otherwise, the interactive text will be added to the corpus as a new corpus text, thereby obtaining a corpus with low duplication.
[0115] In this way, the interactive text is checked for duplicates, a low-repetition corpus is constructed, and the low-repetition corpus is further used as input for training by the existing large language model. In this embodiment, ChatGLM-6B is used as the large language model for training, and the output is the updated large language model of the digital discipline inspection industry, completing the natural language interactive learning of the large language model of the digital discipline inspection industry. Among them, language model training is a common technology in the field of natural language processing.
[0116] Since this embodiment classifies texts, uses similar statistical weights to improve the statistical ability of word segmentation samples of short texts, and uses word segmentation distribution divergence instead of TF-IDF weights, it avoids statistical errors of text length on word segmentation statistical features, improves the Simhash algorithm's ability to check for duplicate content in short texts, and avoids duplicate content checking errors affecting the learning quality of large language models in the digital discipline inspection industry.
[0117] The various embodiments in the present application are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.
[0118] It should be noted that, unless otherwise specified and limited, terms such as "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a circuit structure, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, an element defined by the sentence "including one..." does not exclude the existence of other identical elements in the article or device including the element. In addition, the term "and\or" used herein includes any and all combinations of one or more related listed items.
[0119] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not invented by the present application.
[0120] It will be appreciated that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.
Claims
1. A natural language interactive learning method for a large language model in the digital discipline inspection industry, characterized in that: The method comprises the following steps: Obtain natural language interactive text data of the digital discipline inspection industry and corpus text data in the large prediction model corpus, and perform word segmentation on the text data; All texts are graded according to the number of words in the texts, and the text grade weight of each text grade is determined based on the number of occurrences of each word in each text at each text grade; Determine the similarity statistical weight between each segmentation word and each similar segmentation word at each text level by combining the text level weight of each text level and the difference in the number of characters between each segmentation word and each similar segmentation word at each text level; Each text at each text level is evenly divided into several segments according to the number of words; based on the number of occurrences of each segmentation word and all similar segmentations in each segment in all texts at each text level and the similarity statistical weights between them, a distribution feature vector of each segmentation word at each text level is constructed; Determine the distribution divergence of each segmentation in the interactive text by combining the text level weight differences between the text levels and the differences between the distribution feature vectors of each segmentation in the interactive text and the corresponding segmentation in all text levels; The Simhash code generated for each corpus text and interactive text in the corpus is improved based on distribution divergence, and the large language model of the data discipline inspection industry is trained based on the similarity between the improved Simhash codes.
2. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 1 is characterized in that: The method for determining the text level weight of each text level includes: For each text at each text level, count the number of occurrences of each word segmentation in each text; Count the number of occurrences of each type in each text as the value of the element corresponding to the position in the arrangement result vector; the corresponding position is the value corresponding to the number of occurrences of each type; The text level weight of each text level is determined based on the fractal dimension of the arrangement result vector of all texts under each text level.
3. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 2 is characterized in that: The text level weight of each text level is determined based on the fractal dimension of the arrangement result vector of all texts at each text level. The expression is: Among them, Tl n is the text level weight of the nth text level, M is the number of texts in the nth text level, and Fd m,n It is the fractal dimension of the permutation result vector of the mth text at the nth text level.
4. The natural language interactive learning method for a large language model in the digital discipline inspection industry according to claim 1 is characterized in that: The method for determining the similarity statistical weight between each segmentation and each similar segmentation at each text level includes: For each word at each text level, the words with the same characters are taken as similar words of each word; Determine the character similarity between each segmentation word and each similar segmentation word based on the difference in the number of characters between each segmentation word and each similar segmentation word; The similarity statistical weight between each participle and each similar participle at each text level is determined by combining the text level weight of each text level and the character similarity between each participle and each similar participle at each text level.
5. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 4 is characterized in that: The expression of the character similarity between each segmentation and each similar segmentation is: In the formula, sm s,t is the character similarity between the s-th participle and its t-th similar participle, Sn s,t is the number of characters that are common between the s-th participle and its t-th similar participle, Cn t is the number of characters of the tth similar participle of the sth participle, Cn s is the number of characters in the sth word segmentation, min is the minimum function, and max is the maximum function.
6. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 4 is characterized in that: The method combines the text level weight of each text level and the character similarity between each segmentation word and each similar segmentation word at each text level to determine the similarity statistical weight between each segmentation word and each similar segmentation word at each text level, including: Calculate the difference between the number 1 and the text level weight for each text level; The product of the character similarity between each segmentation and each similar segmentation at each text level and the difference is used as the similarity statistical weight between each segmentation and each similar segmentation at each text level.
7. The natural language interactive learning method for a large language model in the digital discipline inspection industry according to claim 1 is characterized in that: The process of constructing the distribution feature vector of each word segmentation at each text level includes: Determine the number of repetitions of each participle in each paragraph by combining the number of occurrences of each participle in each paragraph with all its similar participles and the similarity statistical weights between them; Constructing a distribution statistics vector for each word segment of each text; the length of the distribution statistics vector is the same as the number of segment segments; The number of repetitions of each word segment in each paragraph in each text is used as the element value of the position of the corresponding paragraph in the distribution statistics vector; The mean vector of the normalized distribution statistical vectors of each word segmentation of all texts at each text level is used as the distribution feature vector of each word segmentation at each text level.
8. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 7 is characterized in that: The method for determining the number of repetitions of each segment word in each segment includes: Counting the number of occurrences of each participle in each paragraph; counting the number of occurrences of each similar participle to each participle in each paragraph; Calculate the number of occurrences of each similar participle of each participle in each paragraph and the product of the similarity statistical weight between each participle and each similar participle, and record it as the first product; Calculating the cumulative sum of the first products of all similar participles of each participle in each segment; The sum of the accumulated sum and the number of occurrences of each word segment in each paragraph is used as the number of repetitions of each word segment in each paragraph.
9. The natural language interactive learning method for a large language model in the digital discipline inspection industry as claimed in claim 1, characterized in that: The expression of the distribution divergence of each word segmentation in the interactive text is: Among them, FS s is the distribution divergence of the s-th word segmentation in the interactive text, It is the operation of rounding up the number in the brackets. N is the maximum text level. DS n,s is the difference between the distribution feature vector of the sth word segmentation in the interactive text and the corresponding word segmentation in the nth text level in the corpus, norm is the normalization function, Tl n is the text level weight of the nth text level, Tl h is the text level weight of the text level where the interactive text is located, and μ is the preset correction parameter.
10. The natural language interactive learning method for a large language model in the digital discipline inspection industry according to claim 1, characterized in that: The method improves the Simhash code generated by each corpus text and interactive text in the corpus based on the distribution divergence, and trains the large language model of the discipline inspection industry based on the similarity between the improved Simhash codes, including: The distribution divergence of each word segmentation replaces the TF-IDF weight in the Simhash algorithm, and a corresponding improved Simhash code is generated for each corpus text and interactive text in the corpus; The corpus is adjusted according to the similarity between each corpus text and the improved Simhash code of the interactive text, and the adjusted corpus is used as the input of the large language model for training, and the updated digital discipline inspection industry large language model is output; When the similarity is less than a preset threshold, the interactive text is added to the corpus; otherwise, no processing is performed.
Citation Information
Cited By
Text generation method and device, electronic equipment and storage medium
CN121745051A
