Sharding method, system and device based on information entropy and Transformer

Abstract

The present invention discloses a fragmentation method, system, and device based on information entropy and Transformer. The method comprises the following steps: calculating a basic surprise index and a corresponding attention distribution entropy value for each word position in a text sequence to be fragmented; dynamically adjusting the basic surprise index using the attention distribution entropy value to obtain a modified surprise index and a set of candidate boundary positions; calculating the KL divergence of each candidate boundary position and obtaining a local entropy increase feature using the information entropy change; fusing the modified surprise index, the local entropy increase feature, and the KL divergence based on an adaptive weighting method to obtain a boundary score for each candidate boundary position; constructing an initial boundary position set, constraining and optimizing the candidate boundary position set to obtain an optimized boundary position set; and segmenting the text sequence to be fragmented to obtain text segments that conform to semantic logic. The fragmentation method proposed by the present invention exhibits good adaptability to corpora from different fields.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a sharding method, system, and device based on information entropy and Transformer. Background Art

[0002] With the surge in the use of large-scale knowledge bases and long text documents, segmenting them into appropriate segments for retrieval, analysis, and model training has become a critical issue. However, traditional segmentation methods, mostly based on fixed lengths or simple rules, are difficult to adapt to content changes and can result in inaccurate segment boundaries, impacting the performance of downstream tasks. For example, using fixed-length segmentation in language models can produce inconsistent boundaries and uneven information content. Research has shown that fixed-length hard segmentation can reduce language model perplexity because the length distribution of semantic units in natural language is uneven, and information content is often evenly distributed along semantic units. Therefore, improper segmentation disrupts the original semantic structure and makes information within the segment incoherent. Furthermore, fixed-length segmentation is prone to cutting off text at non-semantic boundaries, resulting in distorted segment information and affecting subsequent vectorization recall performance. Paragraph segmentation struggles with extremely long text, and format-specific segmentation (such as tables and titles) is unsuitable for diverse document types.

[0003] Furthermore, during human reading, sudden changes in information entropy or surprise often correspond to paragraph or topic transitions. For example, studies have found that entropy peaks in language models closely correspond to actual word boundaries. This suggests the feasibility of using information-theoretic metrics to determine segmentation boundaries. However, existing segmentation methods based on information entropy are still relatively rudimentary and fail to fully integrate the capabilities of modern deep learning models, resulting in poor segmentation results. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the present invention provides a sharding method, system and device based on information entropy and Transformer.

[0005] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0006] A sharding method based on information entropy and Transformer, comprising the following steps:

[0007] Based on the Transformer language model, calculate the basic surprise index and corresponding attention distribution entropy value of each word position in the text sequence to be segmented;

[0008] Dynamically adjusting the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and obtaining each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set;

[0009] The KL divergence of each candidate boundary position in the candidate boundary position set is calculated by the KL divergence algorithm, and the local entropy increase feature of each candidate boundary position is obtained by the information entropy change;

[0010] Based on the adaptive weighting method, the modified surprise degree, local entropy increase feature and KL divergence are fused to obtain the boundary score of each candidate boundary position. Among them, the weight parameters corresponding to the modified surprise degree, local entropy increase feature and KL divergence are dynamically adjusted according to the results of constraints and optimization adjustments;

[0011] Based on the boundary scores, the candidate boundary position set is constrained and optimized to obtain the optimized boundary position set;

[0012] The text sequence to be segmented is segmented by optimizing the boundary position set to obtain text segments that conform to semantic logic.

[0013] As an implementable method, the KL divergence of each candidate boundary position in the candidate boundary position set is calculated by the KL divergence algorithm, specifically: based on each candidate boundary position, the first word distribution of the left segment and the second word distribution of the right segment of the boundary position are calculated, and the KL divergence is calculated based on the first word distribution and the second word distribution.

[0014] As an implementation method, calculating the KL divergence of each candidate boundary position in the candidate boundary position set by using the KL divergence algorithm includes the following steps:

[0015] Construct a first sliding window according to a preset window size, and traverse the text sequence to be segmented in sequence according to a preset step size;

[0016] Based on the current first sliding window position, the text sequence in the current first sliding window is divided into a left sequence and a right sequence;

[0017] The word distributions of the left and right sequences are calculated respectively, and the word distributions of the left and right sequences are normalized to obtain the third and fourth word distributions. The KL divergence is calculated based on the third and fourth word distributions.

[0018] As an implementable method, obtaining the local entropy increase feature of each candidate boundary position by using the information entropy change includes the following steps:

[0019] Based on each candidate boundary position, respectively calculate the first word frequency distribution of the segment on the left side of the boundary position and the second word frequency distribution of the segment on the right side of the boundary position, and perform normalization processing on the first word frequency distribution and the second word frequency distribution to obtain a first probability distribution and a second probability distribution;

[0020] The first information entropy of the left segment is calculated based on the first probability distribution, and the second information entropy of the right segment is calculated based on the second probability distribution; the local entropy increase feature of the boundary position is calculated based on the first information entropy and the second information entropy.

[0021] As an implementable method, obtaining the local entropy increase feature of each candidate boundary position by using the information entropy change includes the following steps:

[0022] Build a sliding window based on the preset window size, and traverse the text sequence to be segmented in sequence according to the preset step size;

[0023] Based on the current sliding window position, the text sequence in the window is divided into the first half window and the second half window;

[0024] Calculate the third word frequency distribution of the first half window and the fourth word frequency distribution of the second half window, and normalize the third word frequency distribution and the fourth word frequency distribution respectively to obtain the third probability distribution and the fourth probability distribution;

[0025] A third information entropy of the first half window is calculated based on the third probability distribution, a fourth information entropy of the second half window is calculated based on the fourth probability distribution, and a local entropy increase feature is calculated based on the third information entropy and the fourth information entropy.

[0026] As an implementation method, the boundary score is obtained by the following formula:

[0027]

[0028] in, represents the i-th boundary score, Respectively represent weight parameters, which are obtained by preset or adaptive adjustment, represents the KL divergence, represents the local entropy increase feature, represents the corrected surprise, and t represents the candidate boundary position.

[0029] As an implementable method, constraining and optimizing the candidate boundary position set based on the boundary score includes the following steps:

[0030] A constrained optimization function is constructed based on the boundary score, and the Lagrange multiplier factor in the constrained optimization function is iterated. In each iteration, the satisfaction of the constraint conditions is evaluated and the value of the Lagrange multiplier factor is adjusted accordingly until the constrained optimization function reaches an extreme value, thereby constraining and optimizing the candidate boundary position set.

[0031] The constrained optimization function is expressed as follows:

[0032]

[0033] Among them, λ1 and λ2 represent the dynamically adjustable Lagrange multiplier factors, g1(x) represents the number of fragments constraint, g2(x) represents the length range constraint, and x represents each fragment. represents the i-th boundary score.

[0034] As an implementable method, constraining and optimizing the candidate boundary position set based on the boundary score further includes the following steps:

[0035] Based on the candidate boundary position set, the text sequence to be segmented is divided into several text segments, the perplexity value of each current text segment is calculated, and the perplexity evaluation index is constructed based on the perplexity value;

[0036] When it is detected that the perplexity value of a text segment exceeds a preset threshold, the boundary position selection rule is dynamically adjusted according to the perplexity evaluation index until the optimal boundary position selection rule is obtained, and then the candidate boundary position set is constrained and optimized. The boundary position selection rule includes weight parameters and constraint conditions, and the constraint conditions include at least one or more of the total number of segments, the minimum length threshold or maximum length threshold of a single segment, and the segment length interval range.

[0037] As an implementable method, the calculation of the perplexity value of each current text segment further includes the following steps:

[0038] A regularization term model is constructed to balance the correlation between segment length and perplexity through the regularization term model, wherein the regularization term model is a weighted function of the deviation value of the segment length from a preset reference length.

[0039] As an implementable method, the following steps are also included:

[0040] When the text sequence to be processed is a real-time text data stream with continuous input, the text sequence is scanned based on the time step, and the local entropy increase feature and KL divergence in the current sliding window are dynamically calculated at each potential boundary position;

[0041] A window refresh period is set, and the local entropy increase feature and the KL divergence in the sliding window are reset according to the refresh period to achieve the update of the local entropy increase feature and the KL divergence.

[0042] As an implementable method, after generating the text fragments that conform to the semantic logic, the following steps are further included:

[0043] Constructing a sharding parameter dataset and mapping the sharding parameter dataset to a downstream task, wherein the sharding parameter dataset includes a modified surprise, a KL divergence, and a dynamically adjusted weight parameter, and the downstream task is the next sharding task;

[0044] Obtain a feedback signal of a downstream task, calculate each shard parameter of the shard parameter set in real time according to the feedback signal, and reversely adjust the value corresponding to each shard parameter of the shard parameter set based on the task performance indicator.

[0045] As an implementable method, the following steps are also included:

[0046] The original text or data stream is obtained and preprocessed, wherein the preprocessing includes: cleaning the original text or data stream, normalizing the format, and filtering out noise to generate a text sequence to be segmented.

[0047] A sharding system based on information entropy and Transformer, including:

[0048] The processing and calculation module calculates the basic surprise index and the corresponding attention distribution entropy value of each word position in the text sequence to be segmented based on the Transformer language model;

[0049] a surprise correction module, which dynamically adjusts the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and obtains each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set;

[0050] The information entropy analysis module calculates the KL divergence of each candidate boundary position in the candidate boundary position set through the KL divergence algorithm, and obtains the local entropy increase feature of each candidate boundary position through the information entropy change;

[0051] The scoring and optimization module performs feature fusion processing on the modified surprising degree, local entropy increase feature, and KL divergence based on an adaptive weighting method to obtain a boundary score for each candidate boundary position. The weight parameters corresponding to the modified surprising degree, local entropy increase feature, and KL divergence are dynamically adjusted according to the results of constraints and optimization adjustments. Based on the boundary scores, the set of candidate boundary positions is constrained and optimized to obtain the optimized boundary position set.

[0052] The segmentation processing module segments the text sequence to be segmented by optimizing the boundary position set to obtain text segments that conform to semantic logic.

[0053] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

[0054] A computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0055] The present invention has significant technical effects due to the adoption of the above technical solutions:

[0056] Compared with the simple fragmentation mode of the existing technical solutions, firstly, the information theory metrics used in this invention, such as information entropy and KL divergence, are universal and do not rely on the vocabulary or syntax of a specific field. Therefore, the fragmentation method proposed in this invention shows good adaptability to corpora from different fields.

[0057] Secondly, the boundaries identified by this application using information entropy and surprise often correspond to important locations such as paragraph topic switches and narrative temporal and spatial changes. Integrating the attention mechanism further enables the identification of implicit topic shifts. For example, where a pronoun refers to a different object, the attention weight pattern will shift significantly, allowing the successful detection of these boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0059] Figure 1 It is a schematic diagram of the overall process of the method of the present invention;

[0060] Figure 2 It is a schematic diagram of the overall structure of the system of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0062] The terms "first", "second", etc. in the claims and specification of this application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is merely a way of distinguishing when describing objects with the same properties in the embodiments of this application. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, so that a process, method, system, product or apparatus that includes a series of units is not necessarily limited to those units, but may include other units not expressly listed or inherent to these processes, methods, products or apparatuses.

[0063] Example 1

[0064] A sharding method based on information entropy and Transformer, comprising the following steps:

[0065] S100, based on the Transformer language model, calculate the basic surprise index and the corresponding attention distribution entropy value of each word position in the text sequence to be segmented;

[0066] S200, dynamically adjusting the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and obtaining each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set;

[0067] S300, calculating the KL divergence of each candidate boundary position in the candidate boundary position set by a KL divergence algorithm, and obtaining a local entropy increase feature of each candidate boundary position by using the information entropy change;

[0068] S400, performing feature fusion processing on the modified surprising degree, the local entropy increase feature, and the KL divergence based on an adaptive weighting method to obtain a boundary score for each candidate boundary position, wherein the weight parameters corresponding to the modified surprising degree, the local entropy increase feature, and the KL divergence are dynamically adjusted according to the results of the constraints and optimization adjustment;

[0069] S500: constraining and optimizing the candidate boundary position set based on the boundary score to obtain an optimized boundary position set;

[0070] S600: Segment the text sequence to be segmented using the optimized boundary position set to obtain text segments that conform to semantic logic.

[0071] The Transformer language model is a deep learning model based on the Transformer architecture and is widely used in natural language processing (NLP) tasks such as language modeling, machine translation, and text generation. The Transformer architecture primarily consists of an encoder and a decoder, both of which are stacked with multiple identical layers, each of which incorporates a self-attention mechanism and a feed-forward neural network. The encoder is responsible for processing the input sequence and converting it into a contextual representation. The encoder encodes the input sequence using a multi-head self-attention mechanism and a feed-forward network. The decoder generates a target sequence based on the encoder output and previously generated outputs. The decoder also uses a multi-head self-attention mechanism and a feed-forward network.

[0072] The self-attention mechanism allows the model to simultaneously focus on all positions in the input sequence, thereby capturing long-range dependencies. In the multi-head attention mechanism, the model uses multiple "heads" to focus on different parts of the input, and each head can learn different language patterns. Positional encoding is added to word embeddings to preserve the order of words because the Transformer does not rely on the order of the sequence to process the input.

[0073] Compared to traditional sequence models like RNNs, the Transformer can process input sequences in parallel, significantly improving computational efficiency. Its self-attention mechanism enables the model to effectively capture relationships between distant elements in a sequence, and the Transformer architecture easily scales to larger datasets and more complex tasks. Therefore, this paper selects the Transformer language model and combines it with other algorithms to produce more accurate and semantically logical text fragments.

[0074] In S100, based on the Transformer language model, the basic surprise index and the corresponding attention distribution entropy value of each word position in the text sequence to be segmented are calculated, as follows:

[0075] Calculate the basic surprise level. The Transformer model's self-attention mechanism can capture the strength of the association between elements in the sequence and determine the degree of change in the contextual attention distribution when the current position is used as the segment boundary.

[0076] Basic surprise:

[0077] in, Indicates current location The unit to be predicted, :The model is predicting Contextual information that is dependent on time;

[0078] :The model is based on context The conditional predicted probability of .

[0079] The calculation method of attention distribution entropy H(t) is as follows:

[0080] Combined with the self-attention information of the Transformer model, the position t , extract the previous layer Transformer attention weight vector A(t), and calculate the attention distribution entropy according to the following formula H A ( t ):

[0081]

[0082] in, Indicates location Word pair position The attention weight of the word.

[0083] In S200, the basic surprise index is dynamically adjusted using the attention distribution entropy value to obtain a modified surprise, and based on the modified surprise, each preliminary candidate boundary position is obtained to form a candidate boundary position set, specifically:

[0084] If the value of the attention distribution entropy of the corresponding position is higher than the first threshold, the surprise degree of the corresponding position is increased to obtain a modified surprise degree. If the value of the attention distribution entropy of the corresponding position is lower than the first threshold, the basic surprise degree of the position is lowered or remains unchanged.

[0085] In other words, the attention distribution entropy is normalized into a weight coefficient, and the modified surprise is obtained by subtracting the base surprise from the weight coefficient. This solution uses the attention distribution entropy to automatically filter out false surprise peaks caused by grammatical structures (such as long modifiers). Furthermore, as a dynamic adjustment scheme, it can achieve adaptive adjustment at each boundary position of the text sequence.

[0086] In one embodiment, the basic surprise index is dynamically adjusted based on the attention distribution entropy value to obtain a modified surprise index, as follows:

[0087] Preset attention weights x , calculate the corrected surprise according to the formula ;

[0088] Attention weights can be adjusted according to scenario requirements through preset or adaptive adjustment rules.

[0089] In obtaining each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set, a dynamic threshold method is used, which includes: calculating the modified surprise mean μ and standard deviation σ of all positions in the text sequence, and satisfying S adj > μ + kσ The position of is taken as the candidate boundary, where k Control the sensitivity to obtain a set of high-confidence boundary locations.

[0090] In S300, the KL divergence of each candidate boundary position in the candidate boundary position set is calculated by the KL divergence algorithm, and the local entropy increase feature of each candidate boundary position is obtained by the information entropy change, including the following steps: based on each candidate boundary position, the first word distribution of the left segment of the boundary position and the second word distribution of the right segment are calculated, based on the first word distribution P L and the second word distribution P R Calculate the KL divergence, or build a first sliding window according to the preset window size, and traverse the text sequence to be segmented in order according to the preset step size;

[0091] Based on the current first sliding window position, the text sequence in the current first sliding window is divided into a left sequence and a right sequence;

[0092] Calculate the word distribution of the left sequence and the right sequence respectively, and normalize the word distribution of the left sequence and the right sequence to obtain the third word distribution and the fourth word distribution. L and the fourth word distribution P R Calculate KL divergence.

[0093] Here, KL divergence, by formula D KL (P L ||P R ) is obtained; where P L is the first word distribution or the third word distribution, P R Second word distribution or fourth word distribution.

[0094] If a position is split so that the KL divergence D between the segments KLIf the value of the KL divergence increases significantly, the corresponding boundary is considered to be truncating two segments with significant differences in topic or context, as the probability of a true semantic boundary is higher. When it exceeds the set threshold B, the boundary score at the corresponding position is strengthened. KL divergence can capture statistical differences in vocabulary distribution and is more consistent with semantic boundary analysis than segmentation based solely on position or punctuation. Experiments have shown that this method significantly improves the accuracy of regular segmentation in diverse texts such as news and conversations, and is particularly good at identifying topic transitions and paragraph logical boundaries.

[0095] In one embodiment, local entropy increase feature data is introduced, and the local entropy increase feature of each candidate boundary position is obtained by using the information entropy change, including the following steps:

[0096] Based on each candidate boundary position, respectively calculate the first word frequency distribution of the segment on the left side of the boundary position and the second word frequency distribution of the segment on the right side of the boundary position, and perform normalization processing on the first word frequency distribution and the second word frequency distribution to obtain a first probability distribution and a second probability distribution;

[0097] The first information entropy of the left segment is calculated based on the first probability distribution, and the second information entropy of the right segment is calculated based on the second probability distribution; the local entropy increase feature of the boundary position is calculated based on the first information entropy and the second information entropy, or a sliding window is constructed according to a preset window size, and the text sequence to be segmented is traversed in sequence according to a preset step size; based on the current sliding window position, the text sequence in the window is divided into a front half window and a second half window; the third word frequency distribution of the first half window and the fourth word frequency distribution of the second half window are calculated; the third word frequency distribution and the fourth word frequency distribution are normalized respectively to obtain a third probability distribution and a fourth probability distribution; the third information entropy of the first half window is calculated based on the third probability distribution, the fourth information entropy of the second half window is calculated based on the fourth probability distribution, and the local entropy increase feature is calculated based on the third information entropy and the fourth information entropy.

[0098] The formula for calculating the local entropy increase feature is: ， in, represents the local entropy increase feature, is the first information entropy or the third information entropy, is the second information entropy or the fourth information entropy, if ,like =0.2, indicating a significant information mutation at that location, which can be used as a basis for boundary reinforcement. This method effectively filters global entropy fluctuation noise through local comparison, improving sensitivity to short text (such as Weibo).

[0099] In S400, the modified surprising degree, the local entropy increase feature, and the KL divergence are fused based on an adaptive weighting method to obtain a boundary score for each candidate boundary position. The boundary score is obtained by the following formula: ,in, 、 、 Represent weight parameters respectively. The optimization goal of the text sequence is to select a set of boundaries To maximize the total score ,in, is the boundary position number. 、 、 ), can be used to specifically strengthen key indicators. For example, in news text, the KL divergence weight can be increased to detect obvious topic switches between paragraphs, while in spoken dialogue, the local entropy increase weight parameter can be increased to detect subtle changes in turn-taking. The weight parameters are preset or adaptively changed according to adaptive rules. The weight parameters corresponding to the modified surprise level, local entropy increase feature, and KL divergence are dynamically adjusted based on the results of constraints and optimization adjustments.

[0100] In step S500, based on the boundary scores, the candidate boundary position set is constrained and optimized to obtain an optimized boundary position set. In a specific embodiment, the constraint and optimization adjustment may include multiple methods, such as constructing a constraint optimization function based on the boundary scores, solving the extreme value through the constraint optimization function to achieve the constraint and optimization adjustment. The constraint optimization function is expressed as follows:

[0101] ,

[0102] Where λ1 and λ2 represent dynamically adjustable Lagrangian multipliers, g1(x) represents the fragment number constraint, and g2(x) represents the length range constraint. The Lagrangian multipliers in the constraint optimization function are iterated, and the satisfaction of the constraints is evaluated at each iteration, and the values ​​of λ1 and λ2 are adjusted accordingly. By finding the extreme value of L and iteratively updating the multipliers, the optimal set of boundary positions is obtained.

[0103] The optimization process flexibly balances objective improvement with constraint satisfaction. When constraints are not met, the multiplier factor is adjusted to automatically balance the situation. This approach finds the optimal segmentation strategy that maximizes the objective function while meeting the constraints. The constrained optimization function dynamically adjusts the number and length of segments to avoid over-segmentation or under-segmentation. This is particularly useful when dynamic adjustment of segment length or number is required, such as in customer service conversations where each segment must contain a complete question-answer pair. By adjusting the multiplier factor, the algorithm can adapt to different scenarios in real time and is suitable for online tasks with varying constraints, such as the real-time conversational text data mentioned above.

[0104] One embodiment also discloses a method for solving extrema using dynamic programming. The advantage of dynamic programming is that it guarantees a globally optimal solution and is suitable for precise segmentation within fixed constraints. For example, when segmenting a news article, an equation is used to strictly maximize the semantic coherence score, ensuring that each paragraph is thematically complete. Different solutions can be selectively employed to meet different text requirements, and they can also be combined.

[0105] In addition, in one embodiment, KL divergence and local entropy increase features are calculated, comprising the following steps: for a text sequence of a real-time stream, local entropy increase features and KL divergence of each potential boundary are calculated based on the time step accumulation and the local entropy increase features and KL divergence in the most recent window are regularly refreshed or reset. This scheme has advantages for text data that is updated in real time, such as real-time conversation text.

[0106] Of course, if you are acquiring real-time text data streams, and calculating KL divergence and local entropy increase features, you have:

[0107] When the text sequence to be processed is a real-time text data stream that is continuously input, the text sequence is scanned based on the time step, and the local entropy increase features and KL divergence in the current sliding window are dynamically calculated at each potential boundary position; a window refresh period is set, and the local entropy increase features and KL divergence in the sliding window are reset according to the refresh period to achieve the update of the local entropy increase features and KL divergence.

[0108] By accumulating statistics by time step, the algorithm adapts to real-time stream changes, making it more user-friendly and time-sensitive for real-time data streams. Since real-time stream data updates rapidly, the algorithm optimizes for locality, relying only on the most recent window for calculations, reducing redundancy and memory consumption. Furthermore, statistics within the window are regularly refreshed to prevent historical bias from affecting the accuracy of the current segmentation.

[0109] In addition, in a specific embodiment, constraining and optimizing the candidate boundary position set based on the boundary score further includes the following steps:

[0110] Based on the candidate boundary position set, the text sequence to be segmented is divided into several text segments, the perplexity value of each current text segment is calculated, and the perplexity evaluation index is constructed based on the perplexity value;

[0111] When it is detected that the perplexity value of a text segment exceeds a preset threshold, the boundary position selection rule is dynamically adjusted according to the perplexity evaluation index until the optimal boundary position selection rule is obtained, and then the candidate boundary position set is constrained and optimized. The boundary position selection rule includes weight parameters and constraint conditions, and the constraint conditions include at least one or more of the total number of segments, the minimum length threshold or maximum length threshold of a single segment, and the segment length interval range.

[0112] In addition, the perplexity value of each text segment is calculated to verify the segmentation quality. A regularization term based on the length of the text segment is also introduced to construct a regularization term model. The regularization term model is used to balance the correlation between segment length and perplexity, wherein the regularization term model is a weighted function of the deviation value of the segment length from the preset reference length. For example, a penalty for segment length or a penalty for the number of boundaries is added to prevent excessive segmentation. This can achieve a balance between reducing model perplexity and avoiding overfitting fragmentation, making segmentation more robust.

[0113] In one embodiment, the regularization term model is expressed as: ,in, Indicates relevance, Indicates confusion, represents the coefficient, Represent the length of the text segment and the preset benchmark length respectively. The perplexity can be calculated according to the existing perplexity calculation method, and the coefficient can be adjusted according to the fragmentation situation. This process can exist together or separately with the construction of a constraint optimization function based on the boundary score, and the solution of the extreme value through the constraint optimization function to achieve constraint and optimization adjustment. The verification results and the fragmentation parameters (such as boundary thresholds, merging rules) are recorded in a structured log together, including abnormal fragment warnings (such as sudden increases in perplexity) and optimization suggestions (such as adjusting boundary sensitivity). This solution implements automated quality monitoring to ensure that the fragmentation results meet both boundary rationality and semantic consistency within the fragment. At the same time, since the perplexity index will be unnaturally reduced due to excessive segment division, a regularization term is added to the perplexity target to overcome this problem.

[0114] In one embodiment, the method further includes the following steps: obtaining raw text or data streams and performing preprocessing, wherein the preprocessing includes cleaning, formatting, and noise filtering the raw text or data streams to generate a preprocessed text sequence. Cleaning, normalization, and noise filtering can reduce abnormal fluctuations in surprise levels caused by data noise. Preprocessing can significantly improve the stability of attention distribution entropy, allowing the modified surprise level to more accurately reflect the true semantic boundaries, and is particularly effective for non-standard text.

[0115] In one embodiment, after generating the text segment that conforms to the semantic logic, the method further includes the following steps:

[0116] Construct a sharding parameter dataset and map it to downstream tasks. The sharding parameter dataset includes modified surprise, KL divergence, and dynamically adjusted weight parameters. Obtain feedback from downstream tasks and calculate each sharding parameter in real time based on the feedback. Inversely adjust the values ​​of each sharding parameter in the sharding parameter dataset based on the task performance metrics. Explore optimal parameter combinations using methods such as gradient descent or Monte Carlo sampling.

[0117] More specifically, for example, if the question-answering system frequently gives incorrect answers at a specific sharding boundary, the surprise weight corresponding to that location is automatically reduced, and the contribution of the KL divergence is increased, while regularization constraints are used to prevent overfitting. This process is iterated over multiple task cycles, ultimately adapting the sharding method to specific scenarios and significantly enhancing the domain adaptability of the sharding results.

[0118] Subsequently, the experimental data and comparative solutions of this application were further disclosed, and the above-mentioned embodiments were applied to data from multiple fields, including large-scale knowledge base texts in open fields such as news articles, financial knowledge, and laws and regulations. In addition, cross-domain knowledge bases (such as encyclopedic corpora covering different topics such as investor education and law) were also involved.

[0119] The experiments used a fixed-length sliding window method and a rule-based topic segmentation method as comparisons. Evaluation metrics included perplexity reduction, commonly used segment boundary detection metrics (such as Pk and WindowDiff), and manually inspected segment semantic integrity. Experimental results showed that the proposed dynamic segmentation method outperformed the comparisons across all evaluation metrics. First, the language model perplexity was significantly reduced, with an average reduction of over 15%, indicating more accurate text prediction after segmentation. Second, the Pk and WindowDiff scores decreased by approximately 10%-20% compared to the baseline. This reduction in both metrics resulted in selected boundaries that were closer to the true manually annotated boundaries, resulting in lower rates of missed and false positives. Qualitative analysis showed that boundaries identified using information entropy and surprisingness often correspond to key locations such as paragraph topic switches and narrative temporal and spatial changes. By integrating an attention mechanism, the algorithm was also able to identify subtle topic transitions, such as where a pronoun refers to a different object. While the language model probability did not change significantly, the attention weight pattern shifted significantly, allowing for successful detection of these boundaries—a capability not possessed by traditional methods.

[0120] Example 2

[0121] A sharding system based on information entropy and Transformer, including:

[0122] The processing and calculation module 100 calculates the basic surprise index and the corresponding attention distribution entropy value of each word position in the text sequence to be segmented based on the Transformer language model;

[0123] A surprise correction module 200 dynamically adjusts the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and based on the modified surprise, obtains each preliminary candidate boundary position to form a candidate boundary position set;

[0124] The information entropy analysis module 300 calculates the KL divergence of each candidate boundary position in the candidate boundary position set using the KL divergence algorithm, and obtains the local entropy increase feature of each candidate boundary position based on the information entropy change;

[0125] Scoring and optimization module 400 performs feature fusion processing on the modified surprising degree, local entropy increase feature, and KL divergence based on an adaptive weighting method to obtain a boundary score for each candidate boundary position. The weight parameters corresponding to the modified surprising degree, local entropy increase feature, and KL divergence are dynamically adjusted based on the results of constraint and optimization adjustment. Based on the boundary scores, the set of candidate boundary positions is constrained and optimized to obtain an optimized set of boundary positions.

[0126] The segmentation processing module 500 segments the text sequence to be segmented using the optimized boundary position set to obtain text segments that conform to semantic logic.

[0127] Various changes and modifications can be made without departing from the spirit and scope of the present invention, and all equivalent technical solutions also fall within the scope of the present invention.

[0128] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referenced to each other.

[0129] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] The present invention is described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0131] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0132] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0133] It should be noted that:

[0134] References in this specification to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Therefore, appearances of the phrases "one embodiment" or "an embodiment" in various places throughout this specification do not necessarily refer to the same embodiment.

[0135] Furthermore, it should be noted that the specific embodiments described in this specification may vary in the shapes and names of their components. Any equivalent or simple variations based on the structure, features, and principles described in the patented concept of this invention are included within the scope of protection of this patent. Persons skilled in the art may make various modifications, additions, or substitutions to the described specific embodiments, and these modifications, as long as they do not deviate from the structure of the invention or exceed the scope defined by the claims, shall fall within the scope of protection of this invention.

Claims

1. A sharding method based on information entropy and Transformer, characterized in that: The following steps are involved: Based on the Transformer language model, calculate the basic surprise index and corresponding attention distribution entropy value of each word position in the text sequence to be segmented; Dynamically adjusting the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and obtaining each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set; The KL divergence of each candidate boundary position in the candidate boundary position set is calculated by the KL divergence algorithm, and the local entropy increase feature of each candidate boundary position is obtained by the information entropy change; Based on the adaptive weighting method, the modified surprise degree, local entropy increase feature and KL divergence are fused to obtain the boundary score of each candidate boundary position. Among them, the weight parameters corresponding to the modified surprise degree, local entropy increase feature and KL divergence are dynamically adjusted according to the results of constraints and optimization adjustments; Based on the boundary scores, the candidate boundary position set is constrained and optimized to obtain the optimized boundary position set; The text sequence to be segmented is segmented by optimizing the boundary position set to obtain text segments that conform to semantic logic.

2. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The KL divergence of each candidate boundary position in the candidate boundary position set is calculated by the KL divergence algorithm, specifically: based on each candidate boundary position, the first word distribution of the left segment and the second word distribution of the right segment of the boundary position are calculated, and the KL divergence is calculated based on the first word distribution and the second word distribution.

3. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The step of calculating the KL divergence of each candidate boundary position in the candidate boundary position set by using the KL divergence algorithm includes the following steps: Construct a first sliding window according to a preset window size, and traverse the text sequence to be segmented in sequence according to a preset step size; Based on the current first sliding window position, the text sequence in the current first sliding window is divided into a left sequence and a right sequence; The word distributions of the left and right sequences are calculated respectively, and the word distributions of the left and right sequences are normalized to obtain the third and fourth word distributions. The KL divergence is calculated based on the third and fourth word distributions.

4. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The method of obtaining the local entropy increase feature of each candidate boundary position by the information entropy change includes the following steps: Based on each candidate boundary position, respectively calculate the first word frequency distribution of the segment on the left side of the boundary position and the second word frequency distribution of the segment on the right side of the boundary position, and perform normalization processing on the first word frequency distribution and the second word frequency distribution to obtain a first probability distribution and a second probability distribution; The first information entropy of the left segment is calculated based on the first probability distribution, and the second information entropy of the right segment is calculated based on the second probability distribution; the local entropy increase feature of the boundary position is calculated based on the first information entropy and the second information entropy.

5. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The method of obtaining the local entropy increase feature of each candidate boundary position by the information entropy change includes the following steps: Build a sliding window based on the preset window size, and traverse the text sequence to be segmented in sequence according to the preset step size; Based on the current sliding window position, the text sequence in the window is divided into the first half window and the second half window; Calculate the third word frequency distribution of the first half window and the fourth word frequency distribution of the second half window, and normalize the third word frequency distribution and the fourth word frequency distribution respectively to obtain the third probability distribution and the fourth probability distribution; A third information entropy of the first half window is calculated based on the third probability distribution, a fourth information entropy of the second half window is calculated based on the fourth probability distribution, and a local entropy increase feature is calculated based on the third information entropy and the fourth information entropy.

6. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The boundary score is obtained by the following formula: in, represents the i-th boundary score, Respectively represent weight parameters, which are obtained by preset or adaptive adjustment. represents the KL divergence, represents the local entropy increase feature, represents the corrected surprise, and t represents the candidate boundary position.

7. The sharding method based on information entropy and Transformer according to claim 1 or 6, characterized in that: The constraining and optimizing the candidate boundary position set based on the boundary score includes the following steps: A constrained optimization function is constructed based on the boundary score, and the Lagrange multiplier factor in the constrained optimization function is iterated. In each iteration, the satisfaction of the constraint conditions is evaluated and the value of the Lagrange multiplier factor is adjusted accordingly until the constrained optimization function reaches an extreme value, thereby constraining and optimizing the candidate boundary position set. The constrained optimization function is expressed as follows: Among them, λ1 and λ2 represent the dynamically adjustable Lagrange multiplier factors, g1(x) represents the number of fragments constraint, g2(x) represents the length range constraint, and x represents each fragment. represents the i-th boundary score.

8. The sharding method based on information entropy and Transformer according to claim 7, characterized in that: The constraining and optimizing the candidate boundary position set based on the boundary score further includes the following steps: Based on the candidate boundary position set, the text sequence to be segmented is divided into several text segments, the perplexity value of each current text segment is calculated, and the perplexity evaluation index is constructed based on the perplexity value; When it is detected that the perplexity value of a text segment exceeds a preset threshold, the boundary position selection rule is dynamically adjusted according to the perplexity evaluation index until the optimal boundary position selection rule is obtained, and then the candidate boundary position set is constrained and optimized. The boundary position selection rule includes weight parameters and constraint conditions, and the constraint conditions include at least one or more of the total number of segments, the minimum length threshold or maximum length threshold of a single segment, and the segment length interval range.

9. The sharding method based on information entropy and Transformer according to claim 8, characterized in that: The calculation of the perplexity value of each current text segment further includes the following steps: A regularization term model is constructed to balance the correlation between segment length and perplexity through the regularization term model, wherein the regularization term model is a weighted function of the deviation value of the segment length from a preset reference length.

10. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The following steps are also included: When the text sequence to be processed is a real-time text data stream with continuous input, the text sequence is scanned based on the time step, and the local entropy increase feature and KL divergence in the current sliding window are dynamically calculated at each potential boundary position; A window refresh period is set, and the local entropy increase feature and the KL divergence in the sliding window are reset according to the refresh period to achieve the update of the local entropy increase feature and the KL divergence.

11. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: After generating a text fragment that conforms to semantic logic, the following steps are also included: Constructing a sharding parameter dataset and mapping the sharding parameter dataset to a downstream task, wherein the sharding parameter dataset includes a modified surprise, a KL divergence, and a dynamically adjusted weight parameter, and the downstream task is the next sharding task; Obtain a feedback signal of a downstream task, calculate each shard parameter of the shard parameter set in real time according to the feedback signal, and reversely adjust the value corresponding to each shard parameter of the shard parameter set based on the task performance indicator.

12. The sharding method based on information entropy and Transformer according to claim 1, characterized in that: The following steps are also included: The original text or data stream is obtained and preprocessed, wherein the preprocessing includes: cleaning the original text or data stream, normalizing the format, and filtering out noise to generate a text sequence to be segmented.

13. A sharding system based on information entropy and Transformer, characterized in that: include: The processing and calculation module calculates the basic surprise index and the corresponding attention distribution entropy value of each word position in the text sequence to be segmented based on the Transformer language model; a surprise correction module, which dynamically adjusts the basic surprise index according to the attention distribution entropy value to obtain a modified surprise, and obtains each preliminary candidate boundary position based on the modified surprise to form a candidate boundary position set; The information entropy analysis module calculates the KL divergence of each candidate boundary position in the candidate boundary position set through the KL divergence algorithm, and obtains the local entropy increase feature of each candidate boundary position through the information entropy change; The scoring and optimization module performs feature fusion processing on the modified surprising degree, local entropy increase feature, and KL divergence based on an adaptive weighting method to obtain a boundary score for each candidate boundary position. The weight parameters corresponding to the modified surprising degree, local entropy increase feature, and KL divergence are dynamically adjusted according to the results of constraints and optimization adjustments. Based on the boundary scores, the set of candidate boundary positions is constrained and optimized to obtain the optimized boundary position set. The segmentation processing module segments the text sequence to be segmented by optimizing the boundary position set to obtain text segments that conform to semantic logic.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.

15. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented.

Images