Text processing methods, devices, media and products

Abstract

The present application discloses a text processing method, device, medium and product, which relate to the field of natural language processing technology. By constructing a "balanced entropy" indicator system based on the fusion of word-level entropy, weighted entropy and sentence-level entropy, the information entropy of text paragraphs is calculated from multiple dimensions. By comparing the entropy increase before and after the paragraph expansion, the paragraph segmentation boundary is determined; the truncation of information-intensive content is avoided, the integrity of the paragraph is improved, and the paragraph generation density and coverage requirements in different application scenarios are met. The application has high practicality and popularization.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text processing method, device, medium and product. Background Art

[0002] In the context of artificial intelligence, natural language processing is gradually becoming intelligent and large-scale, and with the popularization of the Internet, text data processing has also become a research focus.

[0003] When currently segmenting text data into paragraphs, existing technologies fail to consider information density distribution and semantic integrity, resulting in low effectiveness of the segmented paragraphs and their inability to be used for subsequent data analysis and other operations.

[0004] Therefore, there is an urgent need for a text processing method that can quantify the information richness of paragraphs to solve the above technical problems. Summary of the Invention

[0005] The present application provides a text processing method, device, medium and product to at least solve the problems in the related art.

[0006] This application provides a text processing method, including:

[0007] Select unit text data in the original text paragraph as candidate text data according to the unit step traversal;

[0008] Add candidate text data to the current paragraph to obtain an extended paragraph. The initial state of the current paragraph is empty.

[0009] Calculate the first equilibrium entropy corresponding to the extended paragraph and the second equilibrium entropy corresponding to the current paragraph according to the first preset formula;

[0010] Determining an entropy increase of the extended paragraph relative to the current paragraph based on the first equilibrium entropy and the second equilibrium entropy;

[0011] Determine whether the first segmentation condition is met based on the entropy increase and the text length of the current paragraph;

[0012] In response to detecting that the first segmentation condition is met, outputting one or more unit text data in the current paragraph as the first paragraph and setting the current paragraph to a pending state;

[0013] In response to not detecting that the first segmentation condition is met, the extended paragraph is updated as the current paragraph until all original text paragraphs are traversed.

[0014] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned text processing methods when executing the computer program.

[0015] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned text processing methods are implemented.

[0016] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned text processing methods when executed by a processor.

[0017] In the text processing method disclosed in the present application, a "balanced entropy" indicator system is constructed based on the fusion of word-level entropy, weighted entropy and sentence-level entropy to calculate the information entropy of text paragraphs from multiple dimensions. By comparing the entropy increase before and after paragraph expansion, the paragraph segmentation boundary is determined; the truncation of information-intensive content is avoided, the integrity of the paragraph is improved, and the paragraph generation density and coverage requirements in different application scenarios are met. It has high practicality and generalizability. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A flowchart of a text processing method provided in an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a text processing method provided in an embodiment of the present application;

[0021] Figure 3 A diagram of the text processing system architecture provided in an embodiment of the present application;

[0022] Figure 4 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0024] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0025] As disclosed in the background technology, in an era of data explosion, how to automatically segment text has become a research focus. Especially in the process of personalized tuning of large language models, an important task in the data preprocessing stage is to segment the original corpus into multiple segments for the subsequent generation and synthesis of question-answer pairs. Existing technologies often use fixed-length segmentation methods, such as dividing the text into multiple paragraphs based on the number of characters, lexicons, or tokens, and then inputting the large language model again to generate the corresponding questions. This solution is simple and effective, and is commonly used to construct fine-tuning datasets in standard formats such as Alpaca (a dataset format based on instruction fine-tuning) and Self-Instruct (a self-generated instruction framework).

[0026] On this basis, in order to enhance semantic coherence, fixed overlapping windows were introduced for text segmentation, whereby several tokens at the end of the previous paragraph would appear at the beginning of the next paragraph. This technical solution alleviated the problem of semantic fragmentation to a certain extent, but it is still essentially a "position-driven" segmentation strategy and does not make any judgment on the actual information density of the text content. In addition, some studies have attempted to divide text based on structural information such as natural paragraphs, periods, or line breaks, and even used topic boundary detection algorithms such as TextTiling to identify topic changes. However, these methods are usually based on heuristic rules such as word frequency distribution similarity and sliding windows of boundary values ​​between paragraphs, and lack the ability to quantify the "information content" of paragraphs.

[0027] Example 1

[0028] The embodiments of the present application provide a text processing method. This application constructs a "balanced entropy" indicator system based on the fusion of word-level entropy, TF-IDF weighted entropy, and sentence-level entropy, and realizes the control of segmentation granularity through information entropy, meeting the paragraph generation density and coverage requirements in different application scenarios. It has high practicality and generalizability. Figure 1 As shown, applying the method disclosed in the embodiment of the present application to divide the text into paragraphs specifically includes:

[0029] S1. Select unit text data in the original text paragraph as candidate text data according to the unit step size traversal, and add the candidate text data to the current paragraph to obtain an extended paragraph.

[0030] Specifically, the above-mentioned original text paragraphs are usually text data from specific fields (such as medicine, finance, and industry), OCR extracted documents, and structured to natural language tables, etc., and the content can be in multiple languages ​​such as Chinese or English. The above-mentioned unit step is a customized unit, which is used to set the sentence length of the text data added to the current paragraph each time, that is, the sentence length of one unit text data. This application selects one unit text data in turn as a candidate text data. Preferably, it can be set as one sentence as a unit step; of course, it can also be set as two sentences as one unit step, and this application does not limit this.

[0031] It should be noted that, in the embodiment of the present application, a paragraph (i.e., the current paragraph) is first constructed, and the initial state of the paragraph is empty; after the candidate text data is added to the current paragraph, the expected extended paragraph is formed, that is, if the current paragraph is S and the candidate text data is i, the extended paragraph can be expressed as S i , S i = S+i. Each time a candidate text data is added to the current paragraph, the balanced entropy of the extended paragraph and the current paragraph needs to be calculated to determine whether the current paragraph needs to be output as the first paragraph. If it does not need to be output, the extended paragraph is updated to the new current paragraph, and candidate text data is continued to be added to the updated current paragraph.

[0032] When obtaining the original text paragraph, in some implementation scenarios, such as Figure 2 As shown, the embodiment of the present application further proposes to pre-process the original text paragraph before selecting the unit text data, specifically including:

[0033] Perform text type verification on the original text paragraph; in response to detecting that the text type verification has passed, perform denoising on the original text paragraph; in response to detecting that the text type verification has failed, trigger a preset conversion operation according to the current text type of the original text paragraph to convert the text type of the original text paragraph. Specifically, the above-mentioned denoising includes but is not limited to processing steps such as unified encoding, removal of special characters, sentence marking, paragraph restoration, wrong word correction, and blank line filtering. The above-mentioned denoising can be performed by applying conventional technical means in this field, and this application will not be expanded here. This application verifies the original text paragraph before dividing the text data in the original text paragraph to determine that the original text paragraph is converted into a natural language text that is easy to process, and further denoises the original text paragraph to ensure the parsability and semantic coherence of the text data in the original text paragraph.

[0034] It should be noted that the text types of the above-mentioned original text paragraphs generally include natural language text, structured text, and optical character text. The above-mentioned text type verification verifies whether the current text type of the original text paragraph is natural language text. If not, the text type verification fails (i.e., fails). If it is natural language text, the text type verification succeeds (i.e., passes).

[0035] After the text verification fails, the embodiment of the present application also proposes to trigger a preset conversion operation to ensure that the text type of the original text paragraph is a natural language text. Specifically, in response to detecting that the original text paragraph is a structured text, the template rule is called to convert the original text paragraph into a natural language text and then determine that the text type verification has passed. Structured text is a table, a list, a database export content, etc., which needs to be converted into a natural language form through specific rules before it can be segmented and entropy value calculated. Among them, the above-mentioned template rules are pre-set rules that are stored in a certain medium for call. For example, by setting the table conversion rules, it is pre-defined how to convert the row and column information of the table into a natural language description, including header processing, data row conversion and merge cell processing, etc.; by setting the list conversion rules, it is pre-defined how to convert the list items into natural language descriptions, including bullet point processing and hierarchical structure processing, etc.; and by setting the database export content conversion rules, it is pre-defined how to convert database records into natural language descriptions, including field name processing, record merging, data formatting, etc. It should be noted that the number of the above-mentioned template rules is set to match the number of types of structured text, covering rules for converting all different types of structured text into natural language descriptions.

[0036] Furthermore, in response to detecting that the original text paragraph is optical character text, the original text paragraph is cleaned and reconstructed to determine whether the text type verification has passed. Optical character text, or OCR text, refers to text data extracted from images or PDF files using optical character recognition technology. Such text often contains problems such as malformed formatting and incomplete semantics, and therefore requires cleaning and reconstructing. The specific cleaning and reconstructing operations are common technical means and will not be elaborated on in this application.

[0037] In addition, this application also proposes setting dictionaries, stop word lists, and corpus statistical parameters for different languages ​​to ensure that the calculation of the balanced entropy index is applicable to different corpus types such as Chinese and English.

[0038] S2. Calculate a first equilibrium entropy corresponding to the extended paragraph and a second equilibrium entropy corresponding to the current paragraph according to a first preset formula.

[0039] Specifically, the first preset formula is H bal ( S )= αHword ( S )+ βH tfidf ( S )+ γH sent ( S );in, S Indicates paragraphs, including extended paragraphs and the current paragraph. H word ( S ) represents the word-level entropy matching the paragraph, H bal ( S ) represents the equilibrium entropy of matching with the paragraph, H tfidf ( S ) represents the weighted entropy of paragraph matching, H sent ( S ) represents the sentence-level entropy matching the paragraph, α 、 β and γ All of them are empirical parameters. It should be noted that the balanced entropy is used to reflect the information integrity and semantic diversity of the paragraph, and is the core basis for paragraph boundary judgment. Generally speaking, when a wide range of vocabulary needs to be covered, the weight of word-level entropy is increased. α , generally between 0.2 and 0.5; when dealing with scenarios that need to focus on core information, such as technical documents and medical reports, increase the weight of TF-IDF entropy β , generally between 0.3 and 0.6; in scenarios where content balance is required, such as when explaining multiple key points, increase the weight of sentence-level entropy. γ , generally takes a value between 0.2 and 0.4. Of course, those skilled in the art can adjust the empirical parameters according to actual scenarios, and this application does not limit this.

[0040] It can be understood that the calculation formulas for the first equilibrium entropy and the second equilibrium entropy are the same, except that the paragraph data introduced are different.

[0041] It's important to note that word-level entropy is calculated using word-level entropy. It measures the richness of vocabulary within a paragraph by counting the probability of each word appearing within the paragraph. A higher word-level entropy value indicates a more discrete distribution of words within the paragraph, and thus a greater diversity.

[0042] Specifically, the method for obtaining word-level entropy includes: assuming a paragraph is S, the set tag contained in it is W, and a word The frequency of occurrence is f(w), and the probability is calculated according to the third preset formula p (w ), the third preset formula is:

[0043] ;

[0044] Further calculate the word level entropy according to the fourth preset formula H word ( S ), the fourth preset formula is:

[0045] .

[0046] The weighted entropy above uses the TF-IDF value as the weight of a word's importance to calculate a weighted value. This is used to measure the distribution of "high-information-value words" within a text paragraph. This emphasizes key semantic information, strengthens the influence of core semantic words within a paragraph, and helps distinguish between texts that are "superficially rich" and "semantically rich." TF-IDF (Term Frequency-Inverse Document Frequency) is a statistical method used to measure the importance of a word to a document and is commonly used for keyword extraction. TF stands for term frequency, and IDF stands for inverse document frequency. Used together, these two methods reflect the representativeness of a word's occurrence within a document.

[0047] Specifically, first calculate each word using the fifth preset formula Weight , where the fifth preset formula is expressed as:

[0048] ;

[0049] The weighted entropy of the paragraph is further calculated according to the sixth preset formula, wherein the sixth preset formula is:

[0050] .

[0051] Sentence-level entropy, as described above, uses sentences as information units and normalizes the importance score of each sentence to calculate information entropy. This measures the balance of sentence content and the concentration of information distribution within a text, thereby characterizing the structural complexity of the sentence level. Higher entropy values ​​indicate relatively balanced structure and content across sentences, facilitating downstream generation of multiple key points.

[0052] Specifically, split a paragraph into a set of sentences { S 1 , S 2 ,... ,S i}, the score of each sentence h( S i) comes from the sentence length or the number of keywords. Specifically, the entropy value p( S i ), wherein the seventh preset formula is expressed as:

[0053] ;

[0054] And the sentence-level entropy of a paragraph text is calculated by the eighth preset formula H sent ( S ), where the eighth preset formula is:

[0055] .

[0056] This application measures the information entropy of paragraph text from different dimensions and comprehensively measures the information density of paragraphs through a "balanced entropy" indicator algorithm based on the fusion of word-level entropy, TF-IDF weighted entropy and sentence-level entropy; it provides an implementation basis for a paragraph segmentation strategy based on "information density drive", greatly improving the accuracy of subsequent paragraph segmentation.

[0057] S3. Determine the entropy increase of the extended paragraph relative to the current paragraph based on the first equilibrium entropy and the second equilibrium entropy. Specifically, the entropy increase can be expressed as the difference between the first equilibrium entropy and the second equilibrium entropy.

[0058] S4. Determine whether the first segmentation condition is met based on the entropy increase and the text length of the current paragraph.

[0059] Specifically, if the entropy increase is less than the information gain saturation value and the text length of the current paragraph is greater than or equal to the second text threshold, then the first segmentation condition is determined to be satisfied; alternatively, if the entropy increase is greater than the information overload preset value and the text length of the current paragraph is greater than or equal to the second text threshold, then the first segmentation condition is determined to be satisfied. The second text threshold is the minimum value for length protection and can preferably be set to 512. It is set by those skilled in the art based on actual scenarios and is not limited in this application.

[0060] Among them, the information gain saturation value is a preset value greater than zero, usually taken as to Magnitude. Information overload preset value H max is the preset maximum information entropy of each segment. Specifically, the information overload preset value H max The setting of needs to balance two key goals: preventing information overload and ensuring semantic integrity. H max Determine this by following these steps:

[0061] Randomly select at least 1000 text paragraphs from representative texts in the target application field, with each paragraph length ranging from 1 to N sentences (N is usually 20); calculate the equilibrium entropy of each paragraph; sort all the obtained equilibrium entropy values ​​in ascending order and take the 95% quantile as the H max The 95th percentile is used to cover the vast majority of normal paragraphs and resist outlier interference. This percentile can also be adjusted according to task requirements.

[0062] S5. In response to detecting that the first segmentation condition is met, outputting one or more unit text data in the current paragraph as the first paragraph and setting the current paragraph to a pending state.

[0063] Specifically, the outputting of one or more unit text data in the current paragraph as the first paragraph and setting the current paragraph to a pending state includes:

[0064] In response to detecting the generation of the first paragraph, the number of units to be overlapped that match the first paragraph is calculated; starting from the end of the first paragraph, one or more unit text data that match the number of units to be overlapped are selected in reverse order to generate overlapping text data; the overlapping text data is added to the current paragraph to set the current paragraph to a pending state. That is, after each available paragraph (i.e., the first paragraph) is output, the text data that needs to be overlapped into the next paragraph is determined as the context prefix of the next paragraph, thereby ensuring semantic continuity. The present application ensures context continuity through a dynamic inter-segment overlap mechanism, significantly improves the quality and semantic consistency between multiple first paragraphs that match the original text paragraph, and solves the problem of cross-segment information loss after segmentation.

[0065] Furthermore, the calculation process of the number of units to be overlapped that match the first paragraph includes:

[0066] A specified number of unit text data is selected from the end of the first paragraph; an average entropy value at the end of the first paragraph is determined based on the specified number of unit text data, the balanced entropy of the first paragraph matching, and a second preset formula; and the number of units to be overlapped in the first paragraph matching is determined based on the average entropy value at the end of the paragraph, an information overload preset value, a maximum allowable number of overlapping units, and a third preset formula.

[0067] The designated number m is generally set to 3-5 sentences at the end of the paragraph, with the standard being to cover a complete small semantic unit, such as a statement of opinion, evidence, and conclusion. In implementation scenarios where condensed technical points are required, the designated number m is preferably set to 3 sentences; in implementation scenarios where logical continuity is more important, the designated number m is preferably set to 5 sentences. Of course, the specific setting standard can be set by those skilled in the art based on actual scenarios, and this application does not impose any restrictions on this.

[0068] The second preset formula is:

[0069] ;

[0070] in, H tail represents the average entropy value at the end of the segment, m Indicates the number of specified units, S 1 Indicates the first paragraph, Indicates the equilibrium entropy of the first paragraph match.

[0071] The third preset formula is: k=ceil ( H tail / H max × K max );in, k Indicates the number of units to be overlapped, H max Indicates the information overload preset value, K max Indicates the maximum number of overlapping units allowed. H tail represents the average entropy value at the end of the segment, ceil is the rounding function.

[0072] In addition, to avoid repeated inclusion of the balance entropy, the entropy weight of the overlapping part in the next paragraph is increased by a penalty coefficient, preferably set to 0.5, thereby improving the efficiency of model reasoning. Of course, it can also be set by those skilled in the art according to actual conditions, and this application does not limit this. Specifically, an overlapping area identifier is set at the head of a new paragraph, and the balance entropy value can be calculated as a separate paragraph. When calculating the balance entropy of the entire paragraph, the following is applied: effective balance entropy value = (penalty coefficient × balance entropy of the overlapping part) + balance entropy of the non-overlapping part.

[0073] S6. In response to not detecting that the first segmentation condition is met, updating the extended paragraph to be the current paragraph until all original text paragraphs are traversed.

[0074] After detecting that the first segmentation condition is met, in order to avoid the difficulty of subsequent analysis caused by excessively long paragraphs, this application also proposes to protect the length of the current paragraph. Specifically:

[0075] Compare the text length of one or more units of text data within the current paragraph with a first text threshold; in response to detecting that the text length is greater than or equal to the first text threshold, output the one or more units of text data within the current paragraph as the first paragraph and set the current paragraph to a pending state. The first text threshold is a preset maximum value for paragraph length protection, preferably set to 4096, and is set by those skilled in the art based on actual circumstances, and is not limited in this application.

[0076] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0077] Example 2

[0078] The embodiment of the present application further proposes to segment the original text paragraphs in combination with the distribution divergence. Specifically, after executing step S6 disclosed in the above embodiment and failing to detect that the first segmentation condition is met, the present application proposes:

[0079] Calculate the distribution divergence of the current paragraph and the extended paragraph; specifically, it can be obtained by calculating the Jensen-Shannon divergence (JS divergence), substituting the current paragraph and the extended paragraph as two probability distributions for calculation. The specific calculation method is conventional means, and this application will not be expanded here. According to the distribution divergence and the preset divergence value, determine whether the second segmentation condition is met; the preset divergence value is determined by factors such as the characteristics of the text data (such as vocabulary richness and paragraph length) and the field involved in the text. In a loose scenario, it is preferably set to any value between 0.2 and 0.3, and in a strict scenario, it is preferably set to any data between 0.1 and 0.15. The specific value is determined by those skilled in the art according to actual conditions.

[0080] In response to detecting that the second segmentation condition is met, one or more unit text data in the current paragraph is output as the first paragraph and the current paragraph is set to a pending state; wherein the setting of the pending state is consistent with the content disclosed in the aforementioned embodiment, and this application will not be expanded here. In response to not detecting that the second segmentation condition is met, the extended paragraph is updated to the current paragraph until the original text paragraphs are traversed; of course, in order to avoid the difficulty of subsequent analysis caused by excessively long paragraphs, after not detecting that the second segmentation condition is met, this application also proposes to protect the length of the current paragraph. The specific steps are consistent with the content disclosed in the aforementioned embodiment, and this application will not be expanded here.

[0081] This application realizes dynamic paragraph segmentation by combining the fused "balanced entropy" indicator and distribution divergence, which can effectively perceive semantic changes and further improve the accuracy of paragraph segmentation.

[0082] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0083] Example 3

[0084] The embodiment of the present application also proposes to use the above method to segment the original paragraph to generate the first paragraph to construct a training data set for the large language model:

[0085] Understandably, the automated generation of synthetic datasets is becoming a mainstream approach in current large-scale model fine-tuning practices, particularly in scenarios where data privacy is restricted or data in specific domains is scarce. This process typically uses private text data as raw text paragraphs as input, generating question-answer pairs through large models for subsequent supervised fine-tuning training. In the era of rapid development of large language models, segmenting the input text into paragraphs is crucial to ensuring the quality of model generation results. Understandingly, different segmentation methods affect the information organization of paragraphs, and thus the model's ability to extract key content for analysis. Therefore, the design of the segmentation strategy not only impacts generation efficiency but also directly influences the quality and coverage of generated answers. Most current mainstream segmentation methods use fixed-length segmentation, roughly truncating the text to a fixed number of words (e.g., 512 words) or a fixed number of tokens. Some also employ a fixed sliding window with overlap strategy to avoid semantic fragmentation. These methods are simple to implement and have engineering feasibility. However, they also have the following drawbacks:

[0086] Failure to Consider Information Density Distribution: Fixed-length segmentation treats all text equally, failing to recognize that some paragraphs are highly information-rich while others are sparsely information-rich. This can easily lead to truncation of important information or waste of generation resources on ineffective paragraphs. Insufficient Semantic Integrity: Both fixed-length segmentation and strategies based on natural paragraph segmentation can result in the splitting of key sentences or the omission of topic sentences, resulting in a lack of necessary contextual support for the generation model and reduced question-answering quality. Weak Structural Adaptability: Fixed rule-based methods are poorly adapted to structured text such as formatted tables, nested long sentences, and list-like descriptions, and are prone to mis-segmenting and mis-blocking, impacting downstream tasks. Coarse Context Control: Fixed overlapping methods fail to dynamically determine the necessity of overlapping paragraphs, often resulting in the repeated input of excessive and redundant context and increasing the generation burden. Inability to Uniformly Assess Paragraph Effectiveness: Existing methods lack a unified evaluation metric to quantify the information richness of paragraphs, making it impossible to determine whether a paragraph is worth generating questions and answers before segmentation. In summary, existing solutions suffer from varying degrees of deficiencies in the intelligence, adaptability, efficiency, and information preservation of paragraph segmentation. The embodiments of the present application provide a method for segmenting original paragraphs using a text processing method, systematically improving the coverage, accuracy, and cost-effectiveness of synthetic data generation, further improving the quality of the generated training dataset, and thus improving the effect of the large language model.

[0087] The embodiment of the present application expands upon the methods disclosed in the first and second embodiments above. After the first and second embodiments have completed the steps of traversing the original text paragraphs, the embodiment further includes:

[0088] Obtain and normalize one or more first paragraphs matching the original text paragraph to obtain one or more second paragraphs; specifically, output the first paragraphs into JSON (JavaScript Object Notation, a lightweight data exchange format) format or TSV (Tab-Separated Values, a plain text format using tabs as delimiters).

[0089] Add preset prompt words to the second paragraph to generate the third paragraph; for example, "Please generate 5 valuable questions and answers based on the following content." Input the third paragraph into the big model to obtain question and answer text data that matches the third paragraph. The collection is matched with the original text paragraph to construct a training data set to train and fine-tune the big model. It is understandable that the training data set finally generated can be used for supervision fine-tuning, instruction tuning, response quality assessment, data quality filtering and other purposes, and has the advantages of traceability, controllability, and clear structure. The division of the first paragraph is consistent with the text processing method disclosed in Example 1 and Example 2, and this application will not repeat it again.

[0090] The embodiment of the present application provides a method for segmenting paragraphs based on "balanced entropy" and then synthesizing data sets. Through a multi-dimensional information entropy evaluation system, dynamic segmentation boundary detection and overlap control mechanism, it avoids the truncation of information-intensive content, improves paragraph integrity, and ensures that the generated questions can cover all valid information points. In addition, it also reduces the risk of generating invalid questions and answers in information-sparse paragraphs, reduces the number of large model calls, and improves the overall question and answer generation efficiency. Based on a more intelligent, accurate and efficient data segmentation strategy, the effectiveness of the training data set finally generated is greatly improved, thereby improving the training effect of the large model.

[0091] The present application embodiment provides a text processing system, such as Figure 3 Shown, including:

[0092] The paragraph preparation module 310 is used to select unit text data in the original text paragraph as candidate text data according to the unit step size traversal;

[0093] The paragraph preparation module 310 is further configured to add candidate text data to the current paragraph to obtain an extended paragraph. The initial state of the current paragraph is empty.

[0094] An information entropy measurement module 320 is configured to calculate a first balanced entropy corresponding to the extended paragraph and a second balanced entropy corresponding to the current paragraph according to a first preset formula;

[0095] The information entropy measurement module 320 is further configured to determine the entropy increase of the extended paragraph relative to the current paragraph based on the first balanced entropy and the second balanced entropy;

[0096] A segmentation determination module 330 is configured to determine whether a first segmentation condition is met based on the entropy increase and the length of the text of the current paragraph;

[0097] The paragraph processing module 340 is configured to, in response to detecting that the first segmentation condition is met, output one or more unit text data in the current paragraph as the first paragraph and set the current paragraph to a pending state;

[0098] The paragraph processing module 340 is further configured to update the extended paragraph as the current paragraph in response to not detecting that the first segmentation condition is met, until all original text paragraphs are traversed.

[0099] An embodiment of the present application further provides an electronic device, comprising: one or more processors; and a memory associated with the one or more processors, the memory being configured to store program instructions. When the program instructions are read and executed by the one or more processors, the program instructions perform the following operations:

[0100] Select unit text data in the original text paragraph as candidate text data according to the unit step traversal;

[0101] Add candidate text data to the current paragraph to obtain an extended paragraph. The initial state of the current paragraph is empty.

[0102] Calculate the first equilibrium entropy corresponding to the extended paragraph and the second equilibrium entropy corresponding to the current paragraph according to the first preset formula;

[0103] Determining an entropy increase of the extended paragraph relative to the current paragraph based on the first equilibrium entropy and the second equilibrium entropy;

[0104] Determine whether the first segmentation condition is met based on the entropy increase and the text length of the current paragraph;

[0105] In response to detecting that the first segmentation condition is met, outputting one or more unit text data in the current paragraph as the first paragraph and setting the current paragraph to a pending state;

[0106] In response to not detecting that the first segmentation condition is met, the extended paragraph is updated as the current paragraph until all original text paragraphs are traversed.

[0107] According to the first preset formula:

[0108] H bal ( S )= αH word ( S )+ βH tfidf ( S )+ γH sent ( S );

[0109] in, S Indicates paragraphs, including extended paragraphs and the current paragraph. H bal ( S ) represents the equilibrium entropy of matching with the paragraph, H word ( S ) represents the word-level entropy matching the paragraph, H tfidf ( S ) represents the weighted entropy of paragraph matching, H sent ( S ) represents the sentence-level entropy matching the paragraph, α 、 β and γ All are empirical parameters.

[0110] In some implementation scenarios, when the program instructions are read and executed by one or more processors, in response to detecting that the first segmentation condition is not satisfied, the program instructions further perform the following operations:

[0111] Calculate the distribution divergence between the current paragraph and the extended paragraph;

[0112] Determine whether the second segmentation condition is met based on the distribution divergence and the preset divergence value;

[0113] In response to detecting that the second segmentation condition is met, outputting one or more unit text data in the current paragraph as a first paragraph and setting the current paragraph to a pending state;

[0114] In response to not detecting that the second segmentation condition is met, the extended paragraph is updated as the current paragraph until all original text paragraphs are traversed.

[0115] In some implementation scenarios, when the program instructions are read and executed by one or more processors, in response to not detecting that the first segmentation condition is satisfied and in response to not detecting that the second segmentation condition is satisfied, the program instructions further perform the following operations:

[0116] Comparing the text length of one or more units of text data in the current paragraph with a first text threshold;

[0117] In response to detecting that the text length is greater than or equal to the first text threshold, one or more unit text data in the current paragraph are output as the first paragraph and the current paragraph is set to a pending state.

[0118] In some implementation scenarios, when the program instructions are read and executed by one or more processors, after traversing the original text paragraph, the program instructions further perform the following operations:

[0119] Obtaining and normalizing one or more first paragraphs matched by the original text paragraphs to obtain one or more second paragraphs;

[0120] Add preset prompt words to the second paragraph to generate the third paragraph;

[0121] Input the third paragraph into the large model to obtain question-answer text data matching the third paragraph;

[0122] The collection is matched with the original text paragraphs to construct a training dataset for training and fine-tuning large models.

[0123] In some implementation scenarios, when the program instructions are read and executed by one or more processors, the program instructions further perform the following operations:

[0124] If the entropy increase is less than the information gain saturation value and the text length of the current paragraph is greater than or equal to the second text threshold, it is determined that the first segmentation condition is met;

[0125] Alternatively, if the entropy increases above the information overload preset value and the text length of the current paragraph is greater than or equal to the second text threshold, it is determined that the first segmentation condition is met.

[0126] In some implementation scenarios, when the program instructions are read and executed by one or more processors, the program instructions further perform the following operations:

[0127] In response to detecting that a first paragraph is generated, calculating the number of units to be overlapped that match the first paragraph;

[0128] Starting from the end of the first paragraph, one or more unit text data that match the number of units to be overlapped are selected in reverse order to generate overlapping text data;

[0129] Add overlapping text data to the current paragraph to set the current paragraph to a pending state.

[0130] In some implementation scenarios, when the program instructions are read and executed by one or more processors, the program instructions further perform the following operations:

[0131] Select the specified number of unit text data from the end of the first paragraph;

[0132] Determine the average entropy value at the end of the first paragraph based on the unit text data of the specified number of units, the balanced entropy of the first paragraph match, and the second preset formula;

[0133] The number of units to be overlapped for matching in the first paragraph is determined according to the average entropy value at the end of the paragraph, the information overload preset value, the maximum allowed number of overlapping units, and a third preset formula.

[0134] The second preset formula is:

[0135] ;

[0136] in, H tail represents the average entropy value at the end of the segment, m Indicates the number of specified units, S 1 Indicates the first paragraph, Indicates the equilibrium entropy of the first paragraph match.

[0137] The third preset formula is: k=ceil ( H tail / H max × K max );

[0138] in, k Indicates the number of units to be overlapped, H max Indicates the information overload preset value, K max Indicates the maximum number of overlapping units allowed. H tail represents the average entropy value at the end of the segment, ceil is the rounding function.

[0139] In some implementation scenarios, when the program instructions are read and executed by one or more processors, before traversing the original text paragraph according to the unit step length to select unit text data as candidate text data, the program instructions further perform the following operations:

[0140] Perform text type verification on the original text paragraph;

[0141] In response to detecting that the text type check passes, performing denoising processing on the original text paragraph;

[0142] In response to detecting that the text type check fails, a preset conversion operation is triggered according to the current text type of the original text paragraph to convert the text type of the original text paragraph.

[0143] In some implementation scenarios, when the program instructions are read and executed by one or more processors, the program instructions further perform the following operations:

[0144] In response to detecting that the original text paragraph is structured text, calling the template rule to convert the original text paragraph into natural language text and then determining that the text type verification has passed;

[0145] In response to detecting that the original text paragraph is optical character text, the original text paragraph is cleaned and reconstructed and then it is determined that the text type verification passes.

[0146] in, Figure 4 The electronic device architecture is shown as an example, and may include a processor 410, a video display adapter 411, a disk drive 412, an input / output interface 413, a network interface 414, and a memory 420. The processor 410, the video display adapter 411, the disk drive 412, the input / output interface 413, the network interface 414, and the memory 420 may be communicatively connected via a bus 430.

[0147] Among them, the processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0148] The memory 420 can be implemented in the form of ROM (Read Only Memory, writable memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 420 can store an operating system 421 for controlling the execution of the electronic device 400, and a basic input and output system (BIOS) 422 for controlling the low-level operations of the electronic device 400. In addition, a web browser 423, a data storage management system 424, and an icon font processing system 425, etc. can also be stored. The above-mentioned icon font processing system 425 can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0149] The input / output interface 413 is used to connect to input / output modules to enable information input and output. The input / output modules can be configured as components within the device (not shown) or externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, and various sensors, while output devices may include a display, speaker, vibrator, indicator light, and the like.

[0150] The network interface 414 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.).

[0151] The bus 430 comprises a pathway for transmitting information between the various components of the device, such as the processor 410 , the video display adapter 411 , the disk drive 412 , the input / output interface 413 , the network interface 414 , and the memory 420 .

[0152] In addition, the electronic device 400 can also obtain information on specific collection conditions from the virtual resource object collection condition information database for use in condition judgment, etc.

[0153] It should be noted that although the above device only shows the processor 410, video display adapter 411, disk drive 412, input / output interface 413, network interface 414, memory 420, bus 430, etc., in the specific implementation process, the device may also include other components necessary for normal execution. In addition, those skilled in the art will understand that the above device may also include only the components necessary to implement the solution of the present application, and does not necessarily include all the components shown in the figure.

[0154] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned text processing method embodiments when run.

[0155] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0156] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned text processing method embodiments are implemented.

[0157] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned text processing method embodiments are implemented.

[0158] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0159] The above is a detailed introduction to a text processing method provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only intended to help understand the method and core ideas of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, several improvements and modifications may be made to the present application, and such improvements and modifications also fall within the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: The method comprises: Select unit text data in the original text paragraph as candidate text data according to the unit step traversal; Add the candidate text data to the current paragraph to obtain an extended paragraph, where the initial state of the current paragraph is empty; Calculating a first equilibrium entropy corresponding to the extended paragraph and a second equilibrium entropy corresponding to the current paragraph according to a first preset formula; determining an entropy increase of the extended paragraph relative to the current paragraph based on the first equilibrium entropy and the second equilibrium entropy; Determining whether a first segmentation condition is met based on the entropy increase and the text length of the current paragraph; In response to detecting that the first segmentation condition is met, outputting one or more unit text data in the current paragraph as a first paragraph and setting the current paragraph to a pending state; In response to not detecting that the first segmentation condition is met, the extended paragraph is updated as the current paragraph until all original text paragraphs are traversed.

2. The method according to claim 1, characterized in that In response to not detecting that the first segmentation condition is not satisfied, the method further includes: Calculating the distribution divergence between the current paragraph and the extended paragraph; Determining whether a second segmentation condition is met based on the distribution divergence and a preset divergence value; In response to detecting that the second segmentation condition is met, outputting one or more unit text data in the current paragraph as a first paragraph and setting the current paragraph to a pending state; In response to not detecting that the second segmentation condition is met, the extended paragraph is updated as the current paragraph until all original text paragraphs are traversed.

3. The method according to claim 2, characterized in that After the response to not detecting that the first segmentation condition is satisfied and the response to not detecting that the second segmentation condition is satisfied, the method further includes: Comparing the text length of one or more unit text data in the current paragraph with a first text threshold; In response to detecting that the text length is greater than or equal to the first text threshold, one or more unit text data in the current paragraph is output as a first paragraph and the current paragraph is set to a pending state.

4. The method according to claim 1, wherein After traversing the original text paragraphs, the method further includes: Obtaining and normalizing one or more first paragraphs that match the original text paragraphs to obtain one or more second paragraphs; Adding preset prompt words to the second paragraph to generate a third paragraph; Inputting the third paragraph into the large model to obtain question-answer text data matching the third paragraph; The question-answer text data is collected and matched with the original text paragraph to construct a training dataset for training and fine-tuning the large model.

5. The method according to claim 1, wherein The first preset formula is: H bal ( S )= αH word ( S )+ βH tfidf ( S )+ γH sent ( S )； in, S Indicates paragraphs, including extended paragraphs and the current paragraph. H bal ( S ) represents the equilibrium entropy of matching with the paragraph, H word ( S ) represents the word-level entropy matching the paragraph, H tfidf ( S ) represents the weighted entropy of paragraph matching, H sent ( S ) represents the sentence-level entropy matching the paragraph, α 、 β and γ All are empirical parameters.

6. The method according to claim 4, characterized in that The determining whether the first segmentation condition is met according to the entropy increase and the text length of the current paragraph includes: If the entropy increase is less than the information gain saturation value and the text length of the current paragraph is greater than or equal to a second text threshold, it is determined that the first segmentation condition is met; Alternatively, if the entropy increases above a preset information overload value and the text length of the current paragraph is greater than or equal to a second text threshold, it is determined that the first segmentation condition is met.

7. The method according to claim 6, characterized in that Outputting the one or more unit text data in the current paragraph as the first paragraph and setting the current paragraph to a pending state includes: In response to detecting that the first paragraph is generated, calculating the number of units to be overlapped that match the first paragraph; Starting from the end of the first paragraph, one or more unit text data matching the number of units to be overlapped are selected in reverse order to generate overlapping text data; The overlapping text data is added to the current paragraph to set the current paragraph to a pending state.

8. The method according to claim 7, characterized in that The calculating the number of units to be overlapped that match the first paragraph includes: Selecting a specified number of unit text data from the end of the first paragraph; Determining an average entropy value at the end of the first paragraph based on the specified number of unit text data, the balanced entropy of the first paragraph match, and a second preset formula; The number of units to be overlapped that are matched by the first paragraph is determined according to the average entropy value at the end of the paragraph, the information overload preset value, the maximum allowed number of overlapping units, and a third preset formula.

9. The method according to claim 8, characterized in that The second preset formula is: ； in, H tail represents the average entropy value at the end of the segment, m Indicates the number of specified units, S 1 Indicates the first paragraph, represents the equilibrium entropy of the first paragraph match.

10. The method according to claim 8, characterized in that The third preset formula is: k=ceil ( H tail / H max × K max ); in, k Indicates the number of units to be overlapped, H max represents the information overload preset value, K max Indicates the maximum number of overlapping units allowed, H tail represents the average entropy value at the end of the segment, ceil is the rounding function.

11. The method according to claim 1, characterized in that Before selecting unit text data in the original text paragraph as candidate text data by traversing according to the unit step length, the method further includes preprocessing the original text paragraph: Performing text type verification on the original text paragraph; In response to detecting that the text type verification passes, performing a denoising process on the original text paragraph; In response to detecting that the text type check fails, a preset conversion operation is triggered according to the current text type of the original text paragraph to convert the text type of the original text paragraph.

12. The method according to claim 11, characterized in that The method of triggering a preset conversion operation according to the current text type of the original text paragraph to convert the text type of the original text paragraph includes: In response to detecting that the original text paragraph is structured text, calling a template rule to convert the original text paragraph into natural language text and then determining that the text type check has passed; In response to detecting that the original text paragraph is optical character text, the original text paragraph is cleaned and reconstructed and then it is determined that the text type verification passes.

13. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the text processing method according to any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the text processing method according to any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the text processing method according to any one of claims 1 to 12 are implemented.

Images