Text processing method and system based on semantic density

Abstract

The invention provides a text processing method and system based on semantic density, which are applied to the technical field of text processing, and the method comprises the following steps: obtaining a target text, and carrying out feature extraction on the target text to obtain a plurality of text features of each paragraph of the target text; calculating a semantic density score of each paragraph according to the text feature of each paragraph; classifying each paragraph according to the semantic density score of each paragraph to obtain a paragraph category of each paragraph; if the paragraph category of the paragraph is high in density, performing block processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain a plurality of text blocks corresponding to the paragraph; and if the paragraph category of the paragraph is low density, processing the paragraph according to the text type of the target text, the paragraph and the semantic density score thereof, and each other paragraph and the semantic density score thereof, so that information splitting and context loss can be avoided, redundant calculation and resource waste are reduced, and the field adaptability is improved.

Description

Technical Field

[0001] This application relates to the technical field of text processing, and more specifically, to a text processing method and system based on semantic density. Background Art

[0002] Retrieval-Augmented Generation (RAG) technology significantly improves the accuracy and factual consistency of generated content by combining large language models (LLMs) with external knowledge bases. Among them, the core process of retrieval-augmented generation technology includes four stages: text chunking, retrieval, context stitching, and generation; text chunking, as a preprocessing link, directly affects subsequent retrieval efficiency and generation quality.

[0003] Traditional RAG systems generally use fixed-length (such as 512 or 1024 tokens) or simple rule-based chunking strategies (such as splitting by paragraphs or headings) to chunk text. For example, the LangChain framework defaults to splitting documents by a fixed number of tokens, and LlamaIndex supports chunking by paragraphs or chapters. However, existing text chunking methods not only suffer from information fragmentation and context loss, but also have problems such as redundant calculations, resource waste, and lack of domain adaptability. Summary of the Invention

[0004] In view of this, this application provides a text processing method and system based on semantic density, aiming to avoid information fragmentation and context loss, reduce redundant calculations and resource waste, and improve domain adaptability.

[0005] The first aspect of this application provides a text processing method based on semantic density, and the method includes:

[0006] Obtain a target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text;

[0007] Calculate the semantic density score of each paragraph according to the text features of each paragraph;

[0008] Classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density;

[0009] If the paragraph category of the paragraph is high density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph;

[0010] If the paragraph category of the paragraph is low density, process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; where the other paragraphs are paragraphs other than the paragraph.

[0011] Optionally, obtain the target text and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text, including;

[0012] Obtain the target text to be processed; where the target text includes multiple paragraphs;

[0013] For each paragraph, input the paragraph into the distilled language model, so that the distilled language model counts the number of entities in the paragraph through named entity recognition, and calculates the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the number of occurrences of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the number of occurrences of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases;

[0014] Among them, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0015] Optionally, calculating the semantic density score of each paragraph according to the text features of each paragraph includes:

[0016] For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature mapped to the [0, 1] interval.

[0017] Optionally, classifying each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph includes:

[0018] For each paragraph, determine whether the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0019] If the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high density;

[0020] If the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low density.

[0021] Optionally, the process of performing chunking on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph includes:

[0022] According to the text type of the target text, filter out the target basic chunk threshold that matches the paragraph from each preset basic chunk threshold;

[0023] Determine the chunking threshold of the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph, and perform initial chunking on the paragraph according to the chunking threshold and a preset expansion window to obtain at least one chunking information; wherein, the chunking information includes at least one break point;

[0024] If the chunking information includes one break point, determine the break point as the target break point of the chunking information;

[0025] If the chunking information includes multiple break points, obtain the text content related to each break point from within the paragraph, and calculate the break point score for each break point according to the text content of each break point;

[0026] According to the break point scores of each break point, filter out the target break point of the chunking information from each break point;

[0027] Chunk the paragraph according to the target break points of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0028] Optionally, the process of processing the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the paragraphs and their semantic density scores of each other includes:

[0029] Identify at least one target other paragraph that has a coherence relationship with the paragraph from each other paragraph;

[0030] According to the paragraph category of each target other paragraph, determine whether there is at least one low-density target other paragraph among each target other paragraph;

[0031] If there is, according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of at least one low-density target other paragraph and its semantic density score, determine whether there is a low-density target other paragraph that meets the merging condition;

[0032] If there is, merge the paragraph and the low-density target other paragraph that meets the merging condition to obtain one text chunk;

[0033] If not, use the paragraph as a text chunk.

[0034] The second aspect of this application provides a text processing system based on semantic density, and the system includes:

[0035] A feature extraction module, configured to obtain a target text and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text;

[0036] A semantic density score calculation module, configured to calculate the semantic density score of each paragraph according to the text features of each paragraph;

[0037] A classification module, configured to classify each paragraph according to the semantic density scores of the paragraphs to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density;

[0038] A high-density paragraph processing module, configured to, if the paragraph category of the paragraph is high density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph;

[0039] A low-density paragraph processing module, configured to, if the paragraph category of the paragraph is low density, process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

[0040] Optionally, the feature extraction module includes:

[0041] A text acquisition unit, configured to acquire a target text to be processed; wherein, the target text includes multiple paragraphs;

[0042] A feature extraction unit, configured to, for each paragraph, input the paragraph into the distilled language model, enable the distilled language model to count the number of entities in the paragraph through named entity recognition, and calculate the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the occurrence times of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the occurrence times of each logical connective; identify the paragraph according to a preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases;

[0043] Among them, the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0044] Optionally, the semantic density scoring calculation module is specifically configured to:

[0045] For each of the paragraphs, use the distilled language model to calculate the eigenvalue of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target eigenvalue of each text feature obtained by mapping the eigenvalue of each text feature to the interval [0, 1].

[0046] Optionally, the classification module includes:

[0047] A judgment unit, configured to judge whether the semantic density score of each paragraph is greater than or equal to a preset semantic density score threshold;

[0048] A first determination unit, configured to determine that the paragraph category of the paragraph is high density if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0049] A second determination unit, configured to determine that the paragraph category of the paragraph is low density if the threshold density score of the paragraph is less than the preset semantic density score threshold.

[0050] The present application provides a text processing method and system based on semantic density, which obtains a target text, extracts features from the target text to obtain multiple text features for each paragraph of the target text; calculates the semantic density score for each paragraph according to the text features of each paragraph; classifies each paragraph according to the semantic density scores of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density; if the paragraph category of a paragraph is high density, the paragraph is processed in blocks according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text blocks corresponding to the paragraph; if the paragraph category of a paragraph is low density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score for each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; when the paragraph belongs to a high-density paragraph, the paragraph is processed in blocks according to the text type of the target text and the semantic density score of the paragraph, which can not only effectively solve the problems of information fragmentation and context loss caused by block division with a fixed length, but also adapt to paragraphs of different text types, thus realizing domain adaptability; when the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph, thus solving the problems of redundant calculation and resource waste caused by processing with a fixed length. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0052] Figure 1 It is a schematic flowchart of a text processing method based on semantic density provided by an embodiment of the present application;

[0053] Figure 2 It is a schematic structural diagram of a text processing system based on semantic density provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present application.

[0055] In the present application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the phrase "including a..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0056] As can be seen from the above background art, traditional RAG systems generally use a fixed length (such as 512 or 1024 tokens) or a chunking strategy based on simple rules (such as splitting by paragraphs, headings) to chunk text.

[0057] However, chunking text by a fixed length may forcefully split high-density key content (such as technical terms, experimental data, legal clauses) into different text chunks. For example, in a technical white paper, if the paragraph describing "distributed architecture design" is split into two chunks, it may result in only the first half being returned during retrieval, losing the core parameters in the second half, and thus generating an incomplete solution. That is to say, chunking by a fixed length will result in problems of information fragmentation and context loss. Also, for legal-type texts, the details of the clauses need to be retained, and for novel-type texts, the narrative coherence needs to be maintained. It can be seen that different types of texts require different chunking methods. Therefore, the existing chunking methods cannot be used for different types of texts, that is, they lack domain adaptability.

[0058] Moreover, for low-density texts (such as narrative background introductions), using small chunks (fixed length) will significantly increase the ineffective retrieval volume, resulting in problems of redundant calculation and resource waste.

[0059] Therefore, the present application provides a text processing method and system based on semantic density. By calculating the semantic density score of each paragraph, it is possible to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph. In the case where the paragraph belongs to a high-density paragraph, according to the text type of the target text and the semantic density score of the paragraph, the paragraph is block-processed, which can not only effectively solve the problems of information fragmentation and context loss caused by block-processing with a fixed length, but also adapt to paragraphs of different text types, thus achieving domain adaptability. In the case where the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph, so as to solve the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0060] See Figure 1 , which shows a schematic flow chart of a text processing method based on semantic density provided by an embodiment of the present application. The text processing method based on semantic density specifically includes the following steps:

[0061] S101: Obtain a target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text.

[0062] In the process of specifically executing step S101, when it is detected that there is a target text that needs to be processed currently, the target text that needs to be processed can be obtained. Among them, the target text includes multiple paragraphs; the target text is preprocessed so as to perform multi-dimensional feature extraction on each paragraph of the preprocessed target text to obtain multiple text features of each paragraph; among them, the multiple text features of the paragraph include: the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph.

[0063] It should be noted that the operations for preprocessing the target text include: performing text cleaning, normalization processing, parsing the preliminary structure of the target text (parsing out the titles, paragraphs, lists, etc. in the target text), and loading the text type and related parameters corresponding to the target text.

[0064] Optionally, the process of obtaining the target text and extracting features from the target text to obtain multiple text features for each paragraph of the target text can be as follows: Obtain the target text to be processed; where the target text includes multiple paragraphs; for each paragraph, input the paragraph into the distilled language model, so that the distilled language model counts the number of entities in the paragraph through named entity recognition and calculates the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the number of occurrences of each logical connective in the paragraph and calculate the logical connection strength of the paragraph according to the number of occurrences of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases; where the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0065] Specifically, for each paragraph, the language model to be trained can be pre-trained using historical texts, and the trained language model can be distilled to obtain the distilled language model, so that after obtaining the target text, the lightweight named entity recognition model can be used through the distilled language model to identify entities such as professional terms, names, and locations in each paragraph of the target text, count the number of each entity, obtain the number of entities in each paragraph, and finally calculate the ratio of the number of entities in each paragraph to the total number of tokens in the paragraph to obtain the entity density of each paragraph.

[0066] The distilled language model is used to count the occurrence probability of each token in each paragraph, and based on the n-gram word frequency distribution, the Shannon entropy (information entropy) of each paragraph is calculated according to the occurrence probability of each token in each paragraph; where the method of calculating the Shannon entropy of the paragraph is shown in formula (1).

[0067] (1)

[0068] Where n is the total number of tokens in the paragraph, and p(xi) is the occurrence probability of the i-th token x in the paragraph within the paragraph.

[0069] It should be noted that the higher the information entropy, the more uniform the information distribution of the paragraph, and the lower the information entropy, the more concentrated the information distribution of the paragraph.

[0070] The distilled language model is used to extract the key phrase set of each paragraph by applying the TextRank or BM25 algorithm (preset core phrase recognition algorithm), where the key phrase set includes multiple core phrases in the paragraph; for each paragraph, calculate the ratio of the total number of each core phrase in the paragraph to the total number of tokens in the paragraph to obtain the core phrase coverage rate of the paragraph.

[0071] Pre - construct a logical connective table and the weight coefficient of each logical connective in the logical connection table, so that the distilled language model can count the occurrence times of each logical connective in the paragraph according to the logical connective table, calculate the occurrence frequency of each logical connective according to the occurrence times of each logical connective, and finally calculate the logical connection strength of the paragraph according to the occurrence frequencies of each logical connective and their weight coefficients; among them, the calculation method of the logical connection strength of the paragraph is shown in formula (2).

[0072] (2)

[0073] Among them, Lc is the logical connection strength of the paragraph, y is the occurrence frequency of the j - th logical connective in the paragraph, q is the weight coefficient of the logical connective, and N is the total number of logical connectives in the paragraph.

[0074] It should be noted that the logical connective table can include logical connectives such as "therefore", "however", "based on", etc.

[0075] It should also be noted that the weight coefficient of the logical connective representing the causal relationship is higher than that of the logical connective representing the parallel relationship.

[0076] S102: Calculate the semantic density score of each paragraph according to the text features of each paragraph.

[0077] In the specific process of executing step S102, for each paragraph, after extracting multiple text features of the paragraph, the distilled language model can be further used to calculate the feature value of each text feature, and calculate the semantic density score of the paragraph according to the feature values of each text feature.

[0078] Optionally, the process of calculating the semantic density score of each paragraph according to the text features of each paragraph can specifically be: for each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].[[]END]]

[0079] Specifically, for each paragraph, after extracting multiple text features of the paragraph through the distilled semantic model, the text features can be further input into the normalization layer of the distilled semantic model to calculate the eigenvalue of entity density, the eigenvalue of information entropy, the eigenvalue of logical connection strength, and the eigenvalue of core phrase coverage rate through the distilled semantic model; map the eigenvalue of entity density, the eigenvalue of information entropy, the eigenvalue of logical connection strength, and the eigenvalue of core phrase coverage rate to the interval [0, 1] respectively to obtain the target eigenvalue of entity density, the target eigenvalue of information entropy, the target eigenvalue of logical connection strength, and the target eigenvalue of core phrase coverage rate; finally, perform weighted fusion calculation on the target eigenvalue of entity density, the target eigenvalue of information entropy, the target eigenvalue of logical connection strength, and the target eigenvalue of core phrase coverage rate to obtain the semantic density score of the paragraph.

[0080] In some embodiments, for each text feature, the eigenvalue of the text feature can be mapped to the interval [0, 1] by using Min-Max normalization or z-score standardization to obtain the target eigenvalue of the text feature.

[0081] In some embodiments, the weight coefficient of each text feature can be preset so that after obtaining the target eigenvalue of each text feature, weighted fusion calculation can be performed according to the target eigenvalue of each text feature and its weight coefficient to obtain the semantic density score of the corresponding paragraph; wherein, the semantic density score is a real number between 0 and 1; the calculation method of the semantic density score of the paragraph is shown in formula (3).

[0082] (3)

[0083] Where SDS is the semantic density score, Ed is the entity density, Ie is the information entropy, Lc is the logical connection strength, and Kp is the core phrase coverage rate; are the weight coefficients of entity density, information entropy, logical connection strength, and core phrase coverage rate respectively.

[0084] It should be noted that The initial values of can be set to 0.4, 0.2, 0.2, 0.2 respectively, and the weight coefficients of entity density, information entropy, logical connection strength, and core phrase coverage rate can be set according to actual applications, and the embodiments of the present application do not limit this here.

[0085] Furthermore, in the embodiments of the present application, the weight coefficient of each text feature is also dynamically adjusted through a feedback tuning mechanism. For example, when the recall rate of relevant retrieval is low in the initial stage, the weight of can be increased to 0.5 so as to effectively protect technical terms in the text during subsequent chunking operations.

[0086] S103: Classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph.

[0087] In the embodiment of the present application, corresponding preset semantic density score ranges can be set in advance, so that after obtaining the semantic density scores of each paragraph in the target text, the paragraph category of each paragraph can be further determined according to the semantic density score of each paragraph and the preset semantic density score threshold.

[0088] It should be noted that the paragraph category of a paragraph can be high density or low density.

[0089] Optionally, the process of classifying each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph can be: for each paragraph, determine whether the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold; if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high density; if the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low density.

[0090] It should be noted that the preset semantic density score threshold can be 0.7, that is to say, if the semantic density score of a paragraph is greater than or equal to 0.7, it is determined that the paragraph category of the paragraph is high density; if the semantic density score of the paragraph is less than 0.7, it is determined that the paragraph category of the paragraph is low density.

[0091] S104: For each paragraph, determine whether the paragraph category of the paragraph is high density or low density. If the paragraph category of the paragraph is high density, execute step S105; if the paragraph category of the paragraph is low density, execute step S106.

[0092] S105: According to the text type of the target text and the semantic density score of the paragraph, perform chunking processing on the paragraph to obtain multiple text chunks corresponding to the paragraph.

[0093] In the specific process of executing step S105, when it is determined that the paragraph category of the paragraph is high density, obtain the text type of the target text, and determine the chunking threshold of the paragraph according to the text type and the semantic density score of the paragraph, so as to perform initial chunking on the paragraph according to the chunking threshold of the paragraph and the sliding window boundary optimization mechanism to obtain multiple chunking information of the paragraph, where the chunking information includes at least one break point; if the chunking information includes one break point, determine the break point as the target break point of the chunking information; if the chunking information includes multiple break points, the target break point of the chunking information can be screened out from each break point according to the text content in the paragraph; finally, chunk the paragraph according to the target break points of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0094] Optionally, according to the text type of the target text and the semantic density score of the paragraph, the process of chunking the paragraph to obtain multiple text chunks corresponding to the paragraph can be as follows: according to the text type of the target text, select the target basic chunk threshold that matches the paragraph from each pre-set basic chunk threshold; determine the chunking threshold of the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph, and perform initial chunking on the paragraph according to the chunking threshold and the pre-set expansion window to obtain at least chunking information; wherein, the chunking information includes at least one break point; if the chunking information includes one break point, determine the break point as the target break point of the chunking information; if the chunking information includes multiple break points, obtain the text content related to each break point from within the paragraph, and calculate the break point score for each break point according to the text content of each break point; according to the break point scores of each break point, select the target break point of the chunking information from each break point; and chunk the paragraph according to the target break points of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0095] It should be noted that the text type can be a technical text type, a news text type, etc., and the embodiments of the present application do not limit this here.

[0096] In the embodiments of the present application, a corresponding basic chunk threshold can be pre-set for each text type, so that in the case where the paragraph category of the paragraph is determined to be high density, obtain the text type of the target text to which the paragraph belongs, and select the target basic chunk threshold that matches the text type from each pre-set basic chunk threshold, and finally calculate the chunking threshold of the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph; wherein, the calculation method of the chunking threshold of the paragraph is shown in formula (4).

[0097] (4)

[0098] Wherein, is a pre-set adjustment factor. For example, the adjustment factor can be pre-set to 0.5; SDS is the semantic density score of the paragraph.

[0099] It should be noted that it can be seen from formula (4) that the greater the semantic density score of the paragraph, the smaller the chunking threshold used when chunking.

[0100] It should also be noted that a chunking threshold range corresponding to the high-density paragraph category can be pre-set, and the basic chunk threshold corresponding to each text type is set according to the chunking threshold range. Among them, the basic chunk threshold of each text type is within the chunking threshold range.

[0101] For example, the chunking threshold can be 256 - 512 tokens. The basic chunking threshold corresponding to the technical text type can be set to 521 tokens, and the basic chunking threshold corresponding to the news text type can be set to 256 tokens. The basic chunking threshold corresponding to each text type can be set according to the actual application, and the embodiments of the present application do not limit this here.

[0102] In some embodiments, an extended window can be set for the sliding window in advance, and the sliding window is used to scan the paragraph according to the target basic chunking threshold to obtain at least one chunking boundary, and each chunking boundary is extended according to the extended window to obtain the chunking region corresponding to each chunking boundary; for each chunking region, at least one potential break point is identified from the content corresponding to the chunking region, so as to generate corresponding chunking information according to the identified break points. Among them, the potential break points can be "period", "paragraph separator", "title", etc., and the embodiments of the present application do not limit this here.

[0103] It should be noted that the extended window can be set to ±50 tokens in advance, that is to say, after determining the chunking boundary, 50 tokens can be extended before and after the chunking boundary to obtain the chunking region corresponding to the chunking boundary.

[0104] It should also be noted that using the sliding window can effectively avoid the semantic coherence being damaged by hard cutting; for example, the chunking boundary was originally set at the 512th token, but the next sentence contains a key formula. At this time, through the sliding window and according to the extended window for extension, the key formula can be covered, thus avoiding the semantic coherence being damaged by hard cutting.

[0105] In some embodiments, for each chunking information, in the case where it is determined that the chunking information includes multiple break points, for each break point, the text fragments before and after the break point (the first 150 tokens and the last 150 tokens of the break point) can be extracted from the corresponding paragraph to obtain the text content related to the break point, and the text content related to the break point (the front text frequency band and the back text frequency band of the break point) is input into a pre-trained long short-term memory (LSTM) coherence model in a preset format, so that the pre-trained LSTM coherence model performs coherence calculation according to the text content related to the break point to obtain the coherence break score of the break point.

[0106] It should be noted that the preset format can be [front text || separator mark || back text], where the separator mark is used to indicate the position of the break point. The coherence break score of the break point is a real number between 0 and 1. If the coherence break score of the break point is 0, it means that the text fragments before and after the break point are completely coherent and not suitable for splitting; if the coherence break score of the break point is 1, it means that the text fragments before and after the break point are completely disjoint and suitable for splitting.

[0107] It should also be noted that coherent / incoherent texts can be constructed from a large-scale text corpus and input into the LSTM coherence model to be trained, where the LSTM coherence model to be trained is composed of a bidirectional LSTM and an attention mechanism, and the hidden layer size of the LSTM coherence model to be trained is 256; the LSTM coherence model to be trained extracts the word embedding sequences of the coherent / incoherent texts and is trained with the training objective of determining whether the sentences in the predicted text belong to the same semantic unit, so as to obtain the pre-trained LSTM coherence model.

[0108] In some embodiments, the priority corresponding to each breakpoint type can also be set. For example, the priority of a breakpoint of the paragraph separator type is less than the priority of a breakpoint of the subheading type, and the priority of a breakpoint of the subheading type is less than the priority of a breakpoint of the heading type.

[0109] It should be noted that the entity dependency graph of the target text can be constructed to identify the reference relationships between the entities in the target text, and the priority of each breakpoint can be adjusted according to the reference relationships between the entities.

[0110] For example, if a certain breakpoint divides two highly relevant entities, the priority of this breakpoint can be reduced.

[0111] In the embodiments of the present application, for each breakpoint, after obtaining the coherence break score of the breakpoint, the breakpoint type corresponding to the breakpoint can be further determined, and the priority of the breakpoint can be determined according to the breakpoint type; finally, the breakpoint score of the breakpoint is calculated according to the priority, coherence break score and semantic integrity of the breakpoint; among them, the calculation formula of the breakpoint score of the breakpoint is shown in formula (5).

[0112] (5)

[0113] Among them, is a pre-set weight coefficient.

[0114] It should be noted that the semantic integrity is evaluated by detecting whether the key entities or phrases are segmented in the text fragments before and after the corresponding breakpoint.

[0115] In some embodiments, for each piece of chunk information, after obtaining the break point scores of each break point within the chunk information, it is possible to determine whether there is a break point among the break points whose break point score is greater than a preset safety threshold; if not, the chunk boundary corresponding to the chunk information is used as the target break point of the chunk information, that is, it returns to the initial boundary position, so that the paragraph can be chunked according to the chunk boundary subsequently; if there is, among the break points whose break point scores are greater than the preset safety threshold, the break point with the highest break point score is selected, and the break point with the highest break point score is used as the target break point.

[0116] In summary, in the case where the paragraph belongs to a high-density paragraph, by determining the chunking threshold used for chunking the paragraph based on the semantic density score of the paragraph and the text type of the target text to which it belongs, it is possible to effectively break through the bottleneck of information fragmentation and redundant calculation caused by chunking with a fixed length, and significantly improve the corresponding retrieval recall rate and generation quality; moreover, by introducing a sliding window and an LSTM coherence model to intelligently expand to the nearest semantically complete point near the chunk boundary, it can not only avoid hard cutting from destroying the context logic, but also ensure the complete retention of high-density key content (such as technical terms, legal provisions).

[0117] S106: Process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and each other paragraph, where the other paragraphs are paragraphs other than the paragraph.

[0118] During the specific execution of step S106, in the case where the paragraph is determined to be a low-density paragraph, at least one target other paragraph having a coherence relationship with the paragraph is identified from each other paragraph, where the other paragraphs are paragraphs other than the paragraph; according to the paragraph categories and semantic density scores of each target other paragraph, it is determined whether there is a target other paragraph among the target other paragraphs that satisfies the condition of being merged with the paragraph; if there is, if there is, the paragraph and the low-density target other paragraph that satisfies the merging condition are merged to obtain a text chunk; if there is, the paragraph is used as a text chunk.

[0119] It should be noted that other paragraphs located before and after the paragraph in the target text that have not been merged or chunked can be determined as target other paragraphs having a coherence relationship with the paragraph.

[0120] Optionally, the process of processing a paragraph based on the paragraph and its semantic density score and the semantic density scores of each other paragraph may be as follows: identify at least one target other paragraph that has a coherence relationship with the paragraph from each other paragraph; based on the paragraph category of each target other paragraph, determine whether there is at least one target other paragraph with a low density in each target other paragraph; if so, based on the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of at least one target other paragraph with a low density, determine whether there is a target other paragraph with a low density that meets the merging condition; if so, merge the paragraph and the target other paragraph with a low density that meets the merging condition to obtain a text chunk; if not, use the paragraph as a text chunk.

[0121] It should be noted that the merging condition indicates that the average value of the semantic density scores of the paragraph and at least one target other paragraph with a low density is less than the preset semantic density score threshold, and the size of the text chunk obtained after merging the paragraph and at least one target other paragraph with a low density does not exceed the preset text chunk size matching the text type of the target text.

[0122] It should also be noted that a corresponding preset text chunk size can be set in advance for each text type. For example, the preset text chunk size corresponding to the news text type can be set to 2048 tokens; the preset text chunk size corresponding to the event background description text type can be 1024 tokens. The preset text chunk size corresponding to the text type can be set according to actual applications, and the embodiments of the present application do not limit this here.

[0123] In some embodiments, if there is one target other paragraph with a low density, the average value of the semantic density scores of the paragraph and the target other paragraph with a low density can be calculated. If the average value is less than the preset semantic density score threshold, the size of the text chunk obtained after merging the paragraph and the target other paragraph with a low density can be further calculated, and it can be determined whether the size of the text chunk exceeds the preset text chunk size corresponding to the text type of the target text to which it belongs; if not, it is determined that the target other paragraph with a low density meets the merging condition, and the paragraph is merged with the target other paragraph with a low density; if it exceeds, it is determined that the target other paragraph with a low density does not meet the merging condition, and the paragraph is used as a text chunk.

[0124] If there are multiple target other paragraphs with low density, combine the paragraph with each of the target other paragraphs with low density to obtain multiple paragraph combinations, where a paragraph combination includes the paragraph and at least one target other paragraph with low density; for each paragraph combination, the average value of the semantic density scores of the paragraph and the target other paragraphs with low density within the paragraph combination can be calculated. If the average value is less than the preset semantic density score threshold, the size of the text block obtained after merging the paragraph and the target other paragraphs with low density within the paragraph combination can be further calculated, and it can be determined whether the size of the text block exceeds the preset text block size; if not, it is determined that the paragraph combination meets the merging condition, and the paragraph is merged with the target other paragraphs with low density; if it exceeds, it is determined that the paragraph combination does not meet the merging condition.

[0125] As a preferred method, if there is no paragraph combination that meets the merging condition, the paragraph is used as a text block; if there is only one paragraph combination that meets the merging condition, the paragraph and the target other paragraphs with low density within the paragraph combination that meets the merging condition are merged; if there are multiple paragraph combinations that meet the merging condition, the paragraph combination with the largest text block size can be selected from each of the paragraph combinations that meet the merging condition, and the paragraph and the target other paragraphs with low density within the selected paragraph combination are merged, and other paragraph combinations are excluded.

[0126] The present application provides a text processing method based on semantic density, which includes obtaining a target text, extracting features from the target text to obtain multiple text features for each paragraph of the target text; calculating a semantic density score for each paragraph according to the text features of each paragraph; classifying each paragraph according to the semantic density scores of all paragraphs to obtain the paragraph category of each paragraph; where the paragraph category is high density or low density; if the paragraph category of a paragraph is high density, performing chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph; if the paragraph category of a paragraph is low density, processing the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs; where other paragraphs refer to paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score for each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; when the paragraph belongs to a high-density paragraph, performing chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph can not only effectively solve the problems of information fragmentation and context loss caused by chunking with a fixed length, but also adapt to paragraphs of different text types, thus achieving domain adaptation; when the paragraph belongs to a low-density paragraph, processing the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs can solve the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0127] Based on the above text processing method based on semantic density provided by the embodiments of the present application, correspondingly, the embodiments of the present application also provide a text processing system based on semantic density, as Figure 2 shown, the text processing system based on semantic density includes:

[0128] A feature extraction module 21, configured to obtain a target text and extract features from the target text to obtain multiple text features for each paragraph of the target text;

[0129] A semantic density score calculation module 22, configured to calculate a semantic density score for each paragraph according to the text features of each paragraph;

[0130] A classification module 23, configured to classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph; where the paragraph category is high density or low density;

[0131] A high-density paragraph processing module 24, configured to, if the paragraph category of a paragraph is high density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph;

[0132] The low-density paragraph processing module 25 is configured to process a paragraph if the paragraph category of the paragraph is low density, according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

[0133] The specific principles and execution processes of each unit in the text processing system based on semantic density disclosed in the embodiments of the present application are the same as those of the text processing method based on semantic density disclosed in the embodiments of the present application. For details, reference can be made to the corresponding parts in the text processing method based on semantic density disclosed in the embodiments of the present application, which will not be elaborated here.

[0134] The present application provides a text processing system based on semantic density, which obtains a target text, extracts features of the target text to obtain multiple text features of each paragraph of the target text; calculates the semantic density score of each paragraph according to the text features of each paragraph; classifies each paragraph according to the semantic density scores of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density; if the paragraph category of the paragraph is high density, the paragraph is processed by block according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text blocks corresponding to the paragraph; if the paragraph category of the paragraph is low density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score of each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; when the paragraph belongs to a high-density paragraph, the paragraph is processed by block according to the text type of the target text and the semantic density score of the paragraph, which can not only effectively solve the problems of information fragmentation and context loss caused by block processing with a fixed length, but also adapt to paragraphs of different text types, thus realizing domain adaptability; when the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph, so as to solve the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0135] Optionally, the feature extraction module includes:

[0136] A text acquisition unit, configured to acquire a target text to be processed; wherein, the target text includes multiple paragraphs;

[0137] A feature extraction unit, which is used for each paragraph to input the paragraph into the distilled language model, enabling the distilled language model to identify the number of entities in the paragraph through named entity recognition and calculate the entity density of the paragraph based on the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the number of occurrences of each logical connective in the paragraph and calculate the logical connection strength of the paragraph based on the number of occurrences of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph based on the paragraph and its multiple core phrases.

[0138] Among them, the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0139] Optionally, a semantic density score calculation module, which is specifically used for:

[0140] For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph based on the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].

[0141] Optionally, a classification module, including:

[0142] A first judgment unit, which is used for each paragraph to judge whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold;

[0143] A first determination unit, which is used to determine that the paragraph category of the paragraph is high density if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0144] A second determination unit, which is used to determine that the paragraph category of the paragraph is low density if the threshold density score of the paragraph is less than the preset semantic density score threshold.

[0145] Optionally, a high-density paragraph processing module, including:

[0146] A first screening unit, which is used to screen out the target basic block threshold matching the paragraph from each preset basic block threshold according to the text type of the target text;

[0147] An initial chunking unit, which is used to determine the chunking threshold of the paragraph according to the target basic block threshold and the semantic density score of the paragraph, and perform initial chunking on the paragraph according to the chunking threshold and a preset expansion window to obtain at least one chunking information; wherein, the chunking information includes at least one break point;

[0148] A third determination unit, which is used to determine the break point as the target break point of the chunking information if the chunking information includes one break point;

[0149] A breakpoint scoring calculation unit, configured to, if the chunking information includes multiple breakpoints, obtain the text content related to each breakpoint from the paragraph, and calculate the breakpoint score of each breakpoint according to the text content of each breakpoint;

[0150] A second screening unit, configured to screen out the target breakpoints of the chunking information from each breakpoint according to the breakpoint scores of each breakpoint;

[0151] A chunking unit, configured to chunk the paragraph according to the target breakpoints of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0152] Optionally, the low-density paragraph processing module includes:

[0153] An identification unit, configured to identify at least one target other paragraph having a coherence relationship with the paragraph from each other paragraph;

[0154] A second judgment unit, configured to judge whether there is at least one low-density target other paragraph in each target other paragraph according to the paragraph category of each target other paragraph;

[0155] A third judgment unit, configured to, if any, judge whether there is a low-density target other paragraph that meets the merging condition according to the text type of the target text, the paragraph and its semantic density score, and at least one low-density target other paragraph and its semantic density score;

[0156] A merging unit, configured to, if any, merge the paragraph and the low-density target other paragraph that meets the merging condition to obtain a text chunk;

[0157] A fourth determination unit, configured to, if not, use the paragraph as a text chunk.

[0158] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. The key point of each embodiment is to illustrate the differences from other embodiments. In particular, for a system or a system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the related parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0159] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0160] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0161] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A text processing method based on semantic density, characterized in that The method includes: Obtain a target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text; Calculate the semantic density score of each paragraph according to the text features of each paragraph; Classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density; If the paragraph category of the paragraph is high density, perform chunking on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph; If the paragraph category of the paragraph is low density, process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

2. The method according to claim 1, characterized in that, The obtaining of the target text and the performing of feature extraction on the target text to obtain multiple text features of each paragraph of the target text includes: Obtain a target text to be processed; wherein, the target text includes multiple paragraphs; For each paragraph, input the paragraph into the distilled language model, so that the distilled language model counts the number of entities in the paragraph through named entity recognition, and calculates the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through word frequency distribution; count the occurrence times of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the occurrence times of each logical connective; identify the paragraph according to a preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases; Wherein, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute the multiple text features of the paragraph.

3. The method according to claim 2, characterized in that, The calculating of the semantic density score of each paragraph according to the text features of each paragraph includes: For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].

4. The method according to claim 1, wherein The classifying of each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph includes: For each paragraph, determine whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold; If the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high density; If the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low density.

5. The method according to claim 1, wherein Performing chunking on the paragraph according to the text type of the target text and the semantic density score of the paragraph, to obtain a plurality of text chunks corresponding to the paragraph, including: According to the text type of the target text, screening out a target basic chunk threshold that matches the paragraph from each of the preset basic chunk thresholds; Determining a chunking threshold for the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph, and performing initial chunking on the paragraph according to the chunking threshold and a preset expansion window, to obtain at least one chunking information; wherein, the chunking information includes at least one break point; If the chunking information includes one break point, determining the break point as the target break point of the chunking information; If the chunking information includes multiple break points, obtaining the text content related to each break point from within the paragraph, and calculating a break point score for each break point according to the text content of each break point; According to the break point scores of each of the break points, screening out the target break point of the chunking information from each of the break points; Chunking the paragraph according to the target break points of each of the chunking information, to obtain a plurality of text chunks corresponding to the paragraph.

6. The method according to claim 1, wherein The processing of the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the paragraphs of each of the other paragraphs and their semantic density scores, includes: Identifying at least one target other paragraph that has a coherence relationship with the paragraph from each of the other paragraphs; According to the paragraph categories of each of the target other paragraphs, determining whether there is at least one target other paragraph with a low density among each of the target other paragraphs; If so, according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of at least one target other paragraph with a low density and its semantic density score, determining whether there is a target other paragraph with a low density that meets the merging condition; If so, merging the paragraph and the target other paragraph with a low density that meets the merging condition, to obtain one text chunk; If not, using the paragraph as one text chunk.

7. A text processing system based on semantic density, characterized in that, The system includes: A feature extraction module, configured to obtain a target text, and perform feature extraction on the target text, to obtain a plurality of text features of each paragraph of the target text; A semantic density score calculation module, configured to calculate a semantic density score for each paragraph according to the text features of each paragraph; A classification module, configured to classify each paragraph according to the semantic density scores of each paragraph, to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density, or low density; A high density paragraph processing module, configured to, if the paragraph category of the paragraph is high density, perform chunking on the paragraph according to the text type of the target text and the semantic density score of the paragraph, to obtain a plurality of text chunks corresponding to the paragraph; A low-density paragraph processing module, which is used to process the paragraph if the paragraph category of the paragraph is low density, according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

8. The system according to claim 7, characterized in that, The feature extraction module includes: A text acquisition unit, which is used to acquire a target text to be processed; wherein, the target text includes multiple paragraphs; A feature extraction unit, which is used for each paragraph, input the paragraph into the distilled language model, so that the distilled language model counts the number of entities in the paragraph through named entity recognition, and calculates the entity density of the paragraph according to the number of entities; calculates the information entropy of the paragraph through the word frequency distribution; counts the occurrence times of each logical connective in the paragraph, and calculates the logical connection strength of the paragraph according to the occurrence times of each logical connective; recognizes the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculates the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases; Wherein, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

9. The system according to claim 8, wherein The semantic density score calculation module is specifically used for: For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].

10. The system according to claim 7, characterized in that, The classification module includes: A judgment unit, which is used for each paragraph to judge whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold; A first determination unit, which is used to determine that the paragraph category of the paragraph is high density if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold; A second determination unit, which is used to determine that the paragraph category of the paragraph is low density if the threshold density score of the paragraph is less than the preset semantic density score threshold.

Images