A text processing method and system based on semantic density

Abstract

The present application provides a text processing method and system based on semantic density, which are applied to the field of text processing technology. The method includes: obtaining a target text, and extracting features of the target text to obtain multiple text features of each paragraph of the target text; calculating a semantic density score for each paragraph according to the text features of each paragraph; classifying each paragraph according to the semantic density scores of all paragraphs to obtain the paragraph category of each paragraph; if the paragraph category of a paragraph is high density, performing chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph; if the paragraph category of a paragraph is low density, processing the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs, which can avoid information fragmentation and context loss, reduce redundant calculations and resource waste, and improve domain adaptability.

Description

Technical Field

[0001] This application relates to the technical field of text processing, and more specifically, to a text processing method and system based on semantic density. Background Art

[0002] Retrieval-Augmented Generation (RAG) technology significantly improves the accuracy and factual consistency of generated content by combining large language models (LLMs) with external knowledge bases. Among them, the core process of the retrieval-augmented generation technology includes four stages: text chunking, retrieval, context splicing, and generation; text chunking, as a preprocessing link, directly affects subsequent retrieval efficiency and generation quality.

[0003] Traditional RAG systems generally use fixed lengths (such as 512 or 1024 tokens) or chunking strategies based on simple rules (such as splitting by paragraphs or headings) to chunk text. For example, the LangChain framework defaults to splitting documents by a fixed number of tokens, and LlamaIndex supports chunking by paragraphs or chapters. However, existing text chunking methods not only suffer from information fragmentation and context loss, but also have problems such as redundant calculations, resource waste, and lack of domain adaptability. Summary of the Invention

[0004] In view of this, this application provides a text processing method and system based on semantic density, aiming to avoid information fragmentation and context loss, reduce redundant calculations and resource waste, and improve domain adaptability.

[0005] The first aspect of this application provides a text processing method based on semantic density, and the method includes:

[0006] Obtain the target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text;

[0007] Calculate the semantic density score of each paragraph according to the text features of each paragraph;

[0008] Classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density;

[0009] If the paragraph category of the paragraph is high density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph;

[0010] If the paragraph category of the paragraph is low density, process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; where the other paragraphs are paragraphs other than the paragraph.

[0011] Optionally, obtain the target text, perform feature extraction on the target text, and obtain multiple text features of each paragraph of the target text, including;

[0012] Obtain the target text to be processed; where the target text includes multiple paragraphs;

[0013] For each paragraph, input the paragraph into the distilled language model, so that the distilled language model counts the number of entities in the paragraph through named entity recognition, and calculates the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the number of occurrences of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the number of occurrences of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases;

[0014] Among them, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0015] Optionally, calculating the semantic density score of each paragraph according to the text features of each paragraph includes:

[0016] For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature mapped to the [0, 1] interval.

[0017] Optionally, classifying each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph includes:

[0018] For each paragraph, determine whether the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0019] If the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high density;

[0020] If the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low density.

[0021] Optionally, the block processing of the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain a plurality of text blocks corresponding to the paragraph includes:

[0022] According to the text type of the target text, screen out the target basic block threshold matching the paragraph from each preset basic block threshold;

[0023] Determine the block threshold of the paragraph according to the target basic block threshold and the semantic density score of the paragraph, and perform initial block on the paragraph according to the block threshold and a preset expansion window to obtain at least one block information; wherein, the block information includes at least one break point;

[0024] If the block information includes one break point, determine the break point as the target break point of the block information;

[0025] If the block information includes multiple break points, obtain the text content related to each break point from within the paragraph, and calculate the break point score for each break point according to the text content of each break point;

[0026] According to the break point scores of each break point, screen out the target break point of the block information from each break point;

[0027] Perform block on the paragraph according to the target break points of each block information to obtain a plurality of text blocks corresponding to the paragraph.

[0028] Optionally, the processing of the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the paragraphs and their semantic density scores of each other paragraph includes:

[0029] Identify at least one target other paragraph having a coherent relationship with the paragraph from each other paragraph;

[0030] According to the paragraph category of each target other paragraph, determine whether there is at least one low-density target other paragraph in each target other paragraph;

[0031] If there is, judge whether there is a low-density target other paragraph that meets the merging condition according to the text type of the target text, the paragraph and its semantic density score, and the at least one low-density target other paragraph and its semantic density score;

[0032] If there is, merge the paragraph and the low-density target other paragraph that meets the merging condition to obtain a text block;

[0033] If not, use the paragraph as a text chunk.

[0034] The second aspect of the present application provides a text processing system based on semantic density, and the system includes:

[0035] A feature extraction module, configured to obtain a target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text;

[0036] A semantic density score calculation module, configured to calculate the semantic density score of each paragraph according to the text features of each paragraph;

[0037] A classification module, configured to classify each paragraph according to the semantic density scores of the respective paragraphs to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density;

[0038] A high-density paragraph processing module, configured to, if the paragraph category of the paragraph is high density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph;

[0039] A low-density paragraph processing module, configured to, if the paragraph category of the paragraph is low density, process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

[0040] Optionally, the feature extraction module includes:

[0041] A text acquisition unit, configured to acquire a target text to be processed; wherein, the target text includes multiple paragraphs;

[0042] A feature extraction unit, configured to, for each paragraph, input the paragraph into a distilled language model, enable the distilled language model to count the number of entities in the paragraph through named entity recognition, and calculate the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through word frequency distribution; count the occurrence times of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the occurrence times of each logical connective; identify the paragraph according to a preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases;

[0043] Among them, the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0044] Optionally, the semantic density scoring calculation module is specifically configured to:

[0045] For each of the paragraphs, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].

[0046] Optionally, the classification module includes:

[0047] A judgment unit, configured to judge, for each of the paragraphs, whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold;

[0048] A first determination unit, configured to determine that the paragraph category of the paragraph is high density if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0049] A second determination unit, configured to determine that the paragraph category of the paragraph is low density if the threshold density score of the paragraph is less than the preset semantic density score threshold.

[0050] The present application provides a text processing method and system based on semantic density, which obtains a target text, extracts features of the target text to obtain multiple text features of each paragraph of the target text; calculates a semantic density score for each paragraph according to the text features of each paragraph; classifies each paragraph according to the semantic density scores of all paragraphs to obtain the paragraph category of each paragraph; wherein the paragraph category is high density or low density; if the paragraph category of a paragraph is high density, the paragraph is block-processed according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text blocks corresponding to the paragraph; if the paragraph category of a paragraph is low density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs; wherein the other paragraphs are paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score of each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; when the paragraph belongs to a high-density paragraph, the paragraph is block-processed according to the text type of the target text and the semantic density score of the paragraph, which can not only effectively solve the problems of information fragmentation and context loss caused by block-processing with a fixed length, but also adapt to paragraphs of different text types, thereby realizing domain adaptation; when the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs, thereby solving the problems of redundant calculation and resource waste caused by processing with a fixed length. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0052] Figure 1 It is a schematic flowchart of a text processing method based on semantic density provided by an embodiment of the present application;

[0053] Figure 2 It is a schematic structural diagram of a text processing system based on semantic density provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0055] In the present application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "including one..." does not exclude the existence of additional identical elements in the process, method, article or device including the element.

[0056] As can be seen from the above background technology, traditional RAG systems generally use a fixed length (such as 512 or 1024 tokens) or a chunking strategy based on simple rules (such as splitting by paragraphs, headings) to chunk text.

[0057] However, chunking text by a fixed length may forcefully split high-density key content (such as technical terms, experimental data, legal provisions) into different text chunks. For example, in a technical white paper, if the paragraph describing "distributed architecture design" is split into two chunks, it may result in only the first half being returned during retrieval, losing the core parameters in the second half, and thus generating an incomplete solution. That is to say, chunking by a fixed length will cause problems of information fragmentation and context loss. Moreover, for legal-type texts, the details of the clauses need to be retained, and for novel-type texts, the narrative coherence needs to be maintained. It can be seen that different types of texts require different chunking methods. Therefore, the existing chunking methods cannot be used for different types of texts, that is, they lack domain adaptability.

[0058] Moreover, for low-density texts (such as narrative background introductions), using small chunks (fixed length) will significantly increase the ineffective retrieval volume, resulting in problems of redundant calculation and resource waste.

[0059] Therefore, the present application provides a text processing method and system based on semantic density. By calculating the semantic density score of each paragraph, it is possible to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph. When the paragraph belongs to a high-density paragraph, according to the text type of the target text and the semantic density score of the paragraph, the paragraph is block-processed, which can not only effectively solve the problems of information fragmentation and context loss caused by block-processing with a fixed length, but also adapt to paragraphs of different text types, thus achieving domain adaptation. When the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph, so as to solve the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0060] See Figure 1 , which shows a schematic flowchart of a text processing method based on semantic density provided by an embodiment of the present application. The text processing method based on semantic density specifically includes the following steps:

[0061] S101: Obtain a target text, and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text.

[0062] In the process of specifically executing step S101, when it is detected that there is a target text that needs to be processed currently, the target text that needs to be processed can be obtained. Among them, the target text includes multiple paragraphs; the target text is preprocessed so as to perform multi-dimensional feature extraction on each paragraph of the preprocessed target text to obtain multiple text features of each paragraph; among them, the multiple text features of the paragraph include: the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph.

[0063] It should be noted that the operations for preprocessing the target text include: performing text cleaning, normalization processing, parsing the preliminary structure of the target text (parsing out the titles, paragraphs, lists, etc. in the target text), and loading the text type and related parameters corresponding to the target text.

[0064] Optionally, the process of obtaining the target text and extracting features from the target text to obtain multiple text features for each paragraph of the target text can be as follows: Obtain the target text to be processed; where the target text includes multiple paragraphs; for each paragraph, input the paragraph into the distilled language model, so that the distilled language model identifies the number of entities in the paragraph through named entity recognition and calculates the entity density of the paragraph based on the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the number of occurrences of each logical connective in the paragraph and calculate the logical connection strength of the paragraph based on the number of occurrences of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph based on the paragraph and its multiple core phrases; where the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0065] Specifically, for each paragraph, the language model to be trained can be pre-trained using historical texts, and the trained language model can be distilled to obtain the distilled language model, so that after obtaining the target text, the distilled language model can use a lightweight named entity recognition model to identify entities such as professional terms, names, and locations in each paragraph of the target text, count the number of each entity, obtain the number of entities in each paragraph, and finally calculate the ratio of the number of entities in each paragraph to the total number of tokens in the paragraph to obtain the entity density of each paragraph.

[0066] The distilled language model is used to count the occurrence probability of each token in each paragraph, and based on the n-gram word frequency distribution, calculate the Shannon entropy (information entropy) of each paragraph according to the occurrence probability of each token in each paragraph; where the method of calculating the Shannon entropy of the paragraph is shown in formula (1).

[0067] (1)

[0068] Where n is the total number of tokens in the paragraph, and p(xi) is the occurrence probability of the i-th token x in the paragraph within the paragraph.

[0069] It should be noted that the higher the information entropy, the more uniform the information distribution of the paragraph, and the lower the information entropy, the more concentrated the information distribution of the paragraph.

[0070] The distilled language model applies the TextRank or BM25 algorithm (preset core phrase recognition algorithm) to extract the key phrase set of each paragraph, where the key phrase set includes multiple core phrases in the paragraph; for each paragraph, calculate the ratio of the total number of each core phrase in the paragraph to the total number of tokens in the paragraph to obtain the core phrase coverage rate of the paragraph.

[0071] Pre - construct a logical connective table and the weight coefficients of each logical connective in the logical connection table, so that the distilled language model can count the occurrence times of each logical connective in the paragraph according to the logical connective table, calculate the occurrence frequency of each logical connective according to the occurrence times of each logical connective, and finally calculate the logical connection strength of the paragraph according to the occurrence frequencies of each logical connective and their weight coefficients; among them, the calculation method of the logical connection strength of the paragraph is shown in formula (2).

[0072] (2)

[0073] Where Lc is the logical connection strength of the paragraph, y is the occurrence frequency of the j - th logical connective in the paragraph, q is the weight coefficient of the logical connective, and N is the total number of logical connectives in the paragraph.

[0074] It should be noted that the logical connective table may include logical connectives such as "therefore", "however", "based on", etc.

[0075] It should also be noted that the weight coefficient of the logical connective indicating a causal relationship is higher than that of the logical connective indicating a parallel relationship.

[0076] S102: Calculate the semantic density score of each paragraph according to the text features of each paragraph.

[0077] In the specific process of executing step S102, for each paragraph, after extracting multiple text features of the paragraph, the distilled language model can be further used to calculate the feature value of each text feature, and calculate the semantic density score of the paragraph according to the feature values of each text feature.

[0078] Optionally, the process of calculating the semantic density score of each paragraph according to the text features of each paragraph can specifically be: for each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph according to the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].[[]END]]

[0079] Specifically, for each paragraph, after extracting multiple text features of the paragraph through the distilled semantic model, the respective text features can be further input into the normalization layer of the distilled semantic model to calculate the eigenvalue of entity density, the eigenvalue of information entropy, the eigenvalue of logical connection strength, and the eigenvalue of core phrase coverage rate through the distilled semantic model; map the eigenvalue of entity density, the eigenvalue of information entropy, the eigenvalue of logical connection strength, and the eigenvalue of core phrase coverage rate to the interval [0, 1] respectively to obtain the target eigenvalue of entity density, the target eigenvalue of information entropy, the target eigenvalue of logical connection strength, and the target eigenvalue of core phrase coverage rate; finally, perform weighted fusion calculation on the target eigenvalue of entity density, the target eigenvalue of information entropy, the target eigenvalue of logical connection strength, and the target eigenvalue of core phrase coverage rate to obtain the semantic density score of the paragraph.

[0080] In some embodiments, for each text feature, the eigenvalue of the text feature can be mapped to the interval [0, 1] by means of Min - Max normalization or z - score standardization to obtain the target eigenvalue of the text feature.

[0081] In some embodiments, the weight coefficient of each text feature can be preset, so that after obtaining the target eigenvalue of each text feature, weighted fusion calculation can be performed according to the target eigenvalue of each text feature and its weight coefficient to obtain the semantic density score of the corresponding paragraph; wherein, the semantic density score is a real number between 0 and 1; the calculation method of the semantic density score of the paragraph is shown in formula (3).

[0082] (3)

[0083] Wherein, SDS is the semantic density score, Ed is the entity density, Ie is the information entropy, Lc is the logical connection strength, and Kp is the core phrase coverage rate; are respectively the weight coefficients of entity density, information entropy, logical connection strength, and core phrase coverage rate.

[0084] It should be noted that The initial values of can be set to 0.4, 0.2, 0.2, 0.2 respectively, and the weight coefficients of entity density, information entropy, logical connection strength, and core phrase coverage rate can be set according to actual applications, and the embodiments of the present application do not limit this here.

[0085] Furthermore, in the embodiments of the present application, the weight coefficient of each text feature is also dynamically adjusted through a feedback tuning mechanism. For example, when the recall rate of relevant retrieval is low in the initial stage, the weight of can be increased to 0.5 so as to effectively protect technical terms in the text during subsequent chunking operations.

[0086] S103: Classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph.

[0087] In the embodiment of the present application, corresponding preset semantic density score ranges can be preset, so that after obtaining the semantic density scores of each paragraph in the target text, the paragraph category of each paragraph can be further determined according to the semantic density score of each paragraph and the preset semantic density score threshold.

[0088] It should be noted that the paragraph category of a paragraph can be high density or low density.

[0089] Optionally, the process of classifying each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph can be: for each paragraph, determine whether the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold; if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high density; if the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low density.

[0090] It should be noted that the preset semantic density score threshold can be 0.7, that is to say, if the semantic density score of a paragraph is greater than or equal to 0.7, it is determined that the paragraph category of the paragraph is high density; if the semantic density score of the paragraph is less than 0.7, it is determined that the paragraph category of the paragraph is low density.

[0091] S104: For each paragraph, determine whether the paragraph category of the paragraph is high density or low density. If the paragraph category of the paragraph is high density, execute step S105; if the paragraph category of the paragraph is low density, execute step S106.

[0092] S105: According to the text type of the target text and the semantic density score of the paragraph, perform chunking processing on the paragraph to obtain multiple text chunks corresponding to the paragraph.

[0093] In the specific process of executing step S105, when it is determined that the paragraph category of the paragraph is high density, obtain the text type of the target text, and determine the chunking threshold of the paragraph according to the text type and the semantic density score of the paragraph, so as to perform initial chunking on the paragraph according to the chunking threshold of the paragraph and the sliding window boundary optimization mechanism to obtain multiple chunking information of the paragraph, where the chunking information includes at least one breakpoint; if the chunking information includes one breakpoint, determine the breakpoint as the target breakpoint of the chunking information; if the chunking information includes multiple breakpoints, the target breakpoint of the chunking information can be screened out from each breakpoint according to the text content in the paragraph; finally, chunk the paragraph according to the target breakpoints of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0094] Optionally, according to the text type of the target text and the semantic density score of the paragraph, the process of chunking the paragraph to obtain multiple text chunks corresponding to the paragraph can be as follows: According to the text type of the target text, select the target basic chunk threshold that matches the paragraph from each of the pre-set basic chunk thresholds; determine the chunking threshold of the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph, and perform initial chunking on the paragraph according to the chunking threshold and the pre-set expansion window to obtain at least chunking information; where the chunking information includes at least one break point; if the chunking information includes one break point, determine the break point as the target break point of the chunking information; if the chunking information includes multiple break points, obtain the text content related to each break point from within the paragraph, and calculate the break point score for each break point according to the text content of each break point; according to the break point scores of each break point, select the target break point of the chunking information from each break point; chunk the paragraph according to the target break points of each chunking information to obtain multiple text chunks corresponding to the paragraph.

[0095] It should be noted that the text type can be a technical text type, a news text type, etc., and the embodiments of the present application do not limit it here.

[0096] In the embodiments of the present application, a corresponding basic chunk threshold can be pre-set for each text type, so that when it is determined that the paragraph category of the paragraph is high density, obtain the text type of the target text to which the paragraph belongs, and select the target basic chunk threshold that matches the text type from each of the pre-set basic chunk thresholds, and finally calculate the chunking threshold of the paragraph according to the target basic chunk threshold and the semantic density score of the paragraph; where the calculation method of the chunking threshold of the paragraph is shown in formula (4).

[0097] (4)

[0098] Wherein, is a pre-set adjustment factor. For example, the adjustment factor can be pre-set to 0.5; SDS is the semantic density score of the paragraph.

[0099] It should be noted that it can be seen from formula (4) that the greater the semantic density score of the paragraph, the smaller the chunking threshold used when chunking.

[0100] It should also be noted that a chunking threshold range corresponding to the high-density paragraph category can be pre-set, and the basic chunk threshold corresponding to each text type is set according to the chunking threshold range. Among them, the basic chunk threshold of each text type is within the chunking threshold range.

[0101] For example, the chunking threshold can be 256 - 512 tokens. The base chunking threshold corresponding to the technical text type can be set to 521 tokens, and the base chunking threshold corresponding to the news text type can be set to 256 tokens. The base chunking threshold corresponding to each text type can be set according to actual applications, and this is not limited in the embodiments of the present application.

[0102] In some embodiments, an extended window can be set for the sliding window in advance, and the sliding window is used to scan the paragraph according to the target base chunking threshold to obtain at least one chunking boundary, and each chunking boundary is extended according to the extended window to obtain the chunking region corresponding to each chunking boundary; for each chunking region, at least one potential break point is identified from the content corresponding to the chunking region, so as to generate corresponding chunking information according to the identified break points. Among them, the potential break points can be "period", "paragraph separator", "title", etc., and this is not limited in the embodiments of the present application.

[0103] It should be noted that the extended window can be set to ±50 tokens in advance, that is to say, after determining the chunking boundary, 50 tokens can be extended before and after the chunking boundary to obtain the chunking region corresponding to the chunking boundary.

[0104] It should also be noted that using the sliding window can effectively avoid the semantic coherence being damaged by hard cutting; for example, the chunking boundary was originally set at the 512th token, but the next sentence contains a key formula. At this time, the sliding window can be used to extend according to the extended window to cover the key formula, thereby avoiding the semantic coherence being damaged by hard cutting.

[0105] In some embodiments, for each chunking information, when it is determined that the chunking information includes multiple break points, for each break point, the text fragments before and after the break point (the first 150 tokens and the last 150 tokens of the break point) can be extracted from the corresponding paragraph to obtain the text content related to the break point, and the text content related to the break point (the front text frequency band and the back text frequency band of the break point) is input into a pre-trained long short-term memory (LSTM) coherence model in a preset format, so that the pre-trained LSTM coherence model calculates the coherence according to the text content related to the break point to obtain the coherence break score of the break point.

[0106] It should be noted that the preset format can be [front text || separator mark || back text], where the separator mark is used to indicate the position of the break point. The coherence break score of the break point is a real number between 0 and 1. If the coherence break score of the break point is 0, it means that the text fragments before and after the break point are completely coherent and not suitable for splitting; if the coherence break score of the break point is 1, it means that the text fragments before and after the break point are completely disjoint and suitable for splitting.

[0107] It should also be noted that coherent / incoherent texts can be constructed from a large-scale text corpus and input into the LSTM coherence model to be trained, where the LSTM coherence model to be trained consists of a bidirectional LSTM and an attention mechanism, and the hidden layer size of the LSTM coherence model to be trained is 256; the LSTM coherence model to be trained extracts the word embedding sequences of the coherent / incoherent texts and is trained with the training objective of determining whether the sentences in the predicted text belong to the same semantic unit, so as to obtain a pre-trained LSTM coherence model.

[0108] In some embodiments, the priority corresponding to each breakpoint type can also be set. For example, the priority of the breakpoint of the paragraph separator type is less than the priority of the breakpoint of the subtitle type, and the priority of the breakpoint of the subtitle type is less than the priority of the breakpoint of the title type.

[0109] It should be noted that the entity dependency graph of the target text can be constructed to identify the reference relationships between the entities in the target text, and the priority of each breakpoint can be adjusted according to the reference relationships between the entities.

[0110] For example, if a certain breakpoint divides two highly relevant entities, the priority of this breakpoint can be reduced.

[0111] In the embodiments of the present application, for each breakpoint, after obtaining the coherence break score of the breakpoint, the breakpoint type corresponding to the breakpoint can be further determined, and the priority of the breakpoint can be determined according to the breakpoint type; finally, the breakpoint score of the breakpoint is calculated according to the priority, coherence break score and semantic integrity of the breakpoint; among them, the calculation formula of the breakpoint score of the breakpoint is shown in formula (5).

[0112] (5)

[0113] Among them, is a preset weight coefficient.

[0114] It should be noted that the semantic integrity is evaluated by detecting whether the key entities or phrases are segmented in the text fragments before and after the corresponding breakpoint.

[0115] In some embodiments, for each block of information, after obtaining the break point scores of each break point within the block of information, it is possible to determine whether there is a break point among the break points whose break point score is greater than a preset safety threshold; if not, the block boundary corresponding to the block of information is used as the target break point of the block of information, that is, it returns to the initial boundary position, so that the block of the paragraph can be performed according to the block boundary subsequently; if there is, among the break points whose break point scores are greater than the preset safety threshold, the break point with the highest break point score is selected, and the break point with the highest break point score is used as the target break point.

[0116] In summary, in the case where the paragraph belongs to a high-density paragraph, by using the semantic density score of the paragraph and the text type of the target text to which it belongs to determine the block threshold used for the block operation of the paragraph, it is possible to effectively break through the bottleneck of information fragmentation and redundant calculation caused by fixed-length blocking, and significantly improve the corresponding retrieval recall rate and generation quality; moreover, by introducing a sliding window and an LSTM coherence model to intelligently expand to the nearest semantically complete point near the block boundary, it can not only avoid hard cutting from destroying the context logic, but also ensure the complete retention of high-density key content (such as technical terms, legal provisions).

[0117] S106: Process the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and each other paragraph, where the other paragraphs are paragraphs other than the paragraph.

[0118] In the specific process of executing step S106, in the case where it is determined that the paragraph is a low-density paragraph, at least one target other paragraph having a coherence relationship with the paragraph is identified from each other paragraph, where the other paragraphs are paragraphs other than the paragraph; according to the paragraph category and semantic density score of each target other paragraph, it is determined whether there is a target other paragraph among the target other paragraphs that satisfies the condition of being merged with the paragraph; if there is, the paragraph and the low-density target other paragraph that satisfies the merging condition are merged to obtain a text block; if there is, the paragraph is used as a text block.

[0119] It should be noted that other paragraphs located before and after the paragraph in the target text that have not been merged or blocked can be determined as target other paragraphs having a coherence relationship with the paragraph.

[0120] Optionally, the process of processing a paragraph based on the paragraph and its semantic density score and the semantic density scores of each other paragraph may be as follows: identify at least one target other paragraph that has a coherence relationship with the paragraph from each other paragraph; based on the paragraph category of each target other paragraph, determine whether there is at least one target other paragraph with a low density in each target other paragraph; if so, based on the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of at least one target other paragraph with a low density, determine whether there is a target other paragraph with a low density that meets the merging condition; if so, merge the paragraph and the target other paragraph with a low density that meets the merging condition to obtain a text chunk; if not, use the paragraph as a text chunk.

[0121] It should be noted that the merging condition indicates that the average value of the semantic density scores of the paragraph and at least one target other paragraph with a low density is less than the preset semantic density score threshold, and the size of the text chunk obtained after merging the paragraph and at least one target other paragraph with a low density does not exceed the preset text chunk size matching the text type of the target text.

[0122] It should also be noted that a corresponding preset text chunk size can be set in advance for each text type. For example, the preset text chunk size corresponding to the news text type can be set to 2048 tokens; the preset text chunk size corresponding to the event background description text type can be 1024 tokens. The preset text chunk size corresponding to the text type can be set according to actual applications, and the embodiments of the present application do not limit this here.

[0123] In some embodiments, if there is one target other paragraph with a low density, the average value of the semantic density scores of the paragraph and the target other paragraph with a low density can be calculated. If the average value is less than the preset semantic density score threshold, the size of the text chunk obtained after merging the paragraph and the target other paragraph with a low density can be further calculated, and it can be determined whether the size of the text chunk exceeds the preset text chunk size corresponding to the text type of the target text to which it belongs; if not, it is determined that the target other paragraph with a low density meets the merging condition, and the paragraph is merged with the target other paragraph with a low density; if it exceeds, it is determined that the target other paragraph with a low density does not meet the merging condition, and the paragraph is used as a text chunk.

[0124] If there are multiple target other paragraphs with low density, combine the paragraph with each of the target other paragraphs with low density to obtain multiple paragraph combinations, where a paragraph combination includes the paragraph and at least one target other paragraph with low density; for each paragraph combination, the average value of the semantic density scores of the paragraph and the target other paragraphs with low density within the paragraph combination can be calculated. If the average value is less than the preset semantic density score threshold, the size of the text block obtained by merging the paragraph and the target other paragraphs with low density within the paragraph combination can be further calculated, and it can be determined whether the size of the text block exceeds the preset text block size; if not, it is determined that the paragraph combination meets the merging condition, and the paragraph is merged with the target other paragraphs with low density; if it exceeds, it is determined that the paragraph combination does not meet the merging condition.

[0125] As a preferred method, if there is no paragraph combination that meets the merging condition, the paragraph is taken as a text block; if there is only one paragraph combination that meets the merging condition, the paragraph and the target other paragraphs with low density within the paragraph combination that meets the merging condition are merged; if there are multiple paragraph combinations that meet the merging condition, the paragraph combination with the largest text block size can be selected from each of the paragraph combinations that meet the merging condition, and the paragraph and the target other paragraphs with low density within the selected paragraph combination are merged, and other paragraph combinations are excluded.

[0126] The present application provides a text processing method based on semantic density, which obtains a target text, extracts features from the target text, and obtains multiple text features for each paragraph of the target text; calculates the semantic density score for each paragraph according to the text features of each paragraph; classifies each paragraph according to the semantic density scores of all paragraphs to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density; if the paragraph category of a paragraph is high density, the paragraph is segmented according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text segments corresponding to the paragraph; if the paragraph category of a paragraph is low density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs; wherein, the other paragraphs are paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score for each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; in the case that the paragraph belongs to a high-density paragraph, the paragraph is segmented according to the text type of the target text and the semantic density score of the paragraph, which can not only effectively solve the problems of information fragmentation and context loss caused by block segmentation with a fixed length, but also adapt to paragraphs of different text types, thus realizing domain adaptability; in the case that the paragraph belongs to low density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of all other paragraphs, thereby solving the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0127] Based on the above-mentioned text processing method based on semantic density provided by the embodiments of the present application, correspondingly, the embodiments of the present application also provide a text processing system based on semantic density, as Figure 2 shown, the text processing system based on semantic density includes:

[0128] A feature extraction module 21, configured to obtain a target text, extract features from the target text, and obtain multiple text features for each paragraph of the target text;

[0129] A semantic density score calculation module 22, configured to calculate the semantic density score for each paragraph according to the text features of each paragraph;

[0130] A classification module 23, configured to classify each paragraph according to the semantic density score of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density;

[0131] A high-density paragraph processing module 24, configured to, if the paragraph category of a paragraph is high density, segment the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text segments corresponding to the paragraph;

[0132] The low-density paragraph processing module 25 is configured to process a paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and its semantic density score if the paragraph category of the paragraph is low-density; wherein, the other paragraphs are paragraphs other than the paragraph.

[0133] The specific principles and execution processes of each unit in the text processing system based on semantic density disclosed in the embodiments of the present application are the same as those of the text processing method based on semantic density disclosed in the embodiments of the present application. For details, reference can be made to the corresponding parts of the text processing method based on semantic density disclosed in the embodiments of the present application, which will not be elaborated here.

[0134] The present application provides a text processing system based on semantic density, which obtains a target text, extracts features of the target text to obtain multiple text features of each paragraph of the target text; calculates the semantic density score of each paragraph according to the text features of each paragraph; classifies each paragraph according to the semantic density scores of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high-density or low-density; if the paragraph category of the paragraph is high-density, the paragraph is block-processed according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text blocks corresponding to the paragraph; if the paragraph category of the paragraph is low-density, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and its semantic density score; wherein, the other paragraphs are paragraphs other than the paragraph. It can be seen that the technical solution provided by the present application calculates the semantic density score of each paragraph, so as to evaluate whether the paragraph belongs to a high-density paragraph or a low-density paragraph according to the semantic density score of the paragraph; when the paragraph belongs to a high-density paragraph, the paragraph is block-processed according to the text type of the target text and the semantic density score of the paragraph, which can not only effectively solve the problems of information fragmentation and context loss caused by block-processing with a fixed length, but also adapt to paragraphs of different text types, thereby achieving domain adaptation; when the paragraph belongs to a low-density paragraph, the paragraph is processed according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and its semantic density score, thereby solving the problems of redundant calculation and resource waste caused by processing with a fixed length.

[0135] Optionally, the feature extraction module includes:

[0136] A text acquisition unit, configured to acquire a target text to be processed; wherein, the target text includes multiple paragraphs;

[0137] A feature extraction unit, which is used for each paragraph to input the paragraph into the distilled language model, enabling the distilled language model to count the number of entities in the paragraph through named entity recognition and calculate the entity density of the paragraph based on the number of entities; calculate the information entropy of the paragraph through the word frequency distribution; count the occurrence times of each logical connective in the paragraph and calculate the logical connection strength of the paragraph based on the occurrence times of each logical connective; identify the paragraph according to the preset core phrase recognition algorithm to obtain multiple core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph based on the paragraph and its multiple core phrases.

[0138] Among them, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

[0139] Optionally, a semantic density score calculation module, which is specifically used for:

[0140] For each paragraph, use the distilled language model to calculate the feature value of each text feature of the paragraph, and calculate the semantic density score of the paragraph based on the target feature value of each text feature obtained by mapping the feature value of each text feature to the interval [0, 1].

[0141] Optionally, a classification module, including:

[0142] A first judgment unit, which is used for each paragraph to judge whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold;

[0143] A first determination unit, which is used to determine that the paragraph category of the paragraph is high density if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold;

[0144] A second determination unit, which is used to determine that the paragraph category of the paragraph is low density if the threshold density score of the paragraph is less than the preset semantic density score threshold.

[0145] Optionally, a high-density paragraph processing module, including:

[0146] A first screening unit, which is used to screen out the target basic block threshold matching the paragraph from each preset basic block threshold according to the text type of the target text;

[0147] An initial chunking unit, which is used to determine the chunking threshold of the paragraph according to the target basic block threshold and the semantic density score of the paragraph, and perform initial chunking on the paragraph according to the chunking threshold and the preset expansion window to obtain at least one chunking information; where the chunking information includes at least one break point;

[0148] A third determination unit, which is used to determine the break point as the target break point of the chunking information if the chunking information includes one break point;

[0149] A break point scoring calculation unit, configured to, if the chunking information includes multiple break points, obtain the text content related to each break point from the paragraph, and calculate the break point score of each break point according to the text content of each break point;

[0150] A second screening unit, configured to screen out the target break points of the chunking information from each break point according to the break point scores of each break point;

[0151] A chunking unit, configured to chunk the paragraph according to the target break points of each chunking information, and obtain multiple text chunks corresponding to the paragraph.

[0152] Optionally, the low-density paragraph processing module includes:

[0153] An identification unit, configured to identify at least one target other paragraph having a coherence relationship with the paragraph from each other paragraph;

[0154] A second judgment unit, configured to judge whether there is at least one low-density target other paragraph in each target other paragraph according to the paragraph category of each target other paragraph;

[0155] A third judgment unit, configured to, if any, judge whether there is a low-density target other paragraph that meets the merging condition according to the text type of the target text, the paragraph and its semantic density score, and at least one low-density target other paragraph and its semantic density score;

[0156] A merging unit, configured to, if any, merge the paragraph and the low-density target other paragraph that meets the merging condition to obtain a text chunk;

[0157] A fourth determination unit, configured to, if not, use the paragraph as a text chunk.

[0158] Each embodiment in this specification is described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for a system or a system embodiment, since it is basically similar to the method embodiment, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiment. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0159] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered as exceeding the scope of this application.

[0160] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but rather to the broadest scope consistent with the principles and novel features disclosed herein.

[0161] The above are only the preferred embodiments of this application. It should be noted that for those of ordinary skill in the art, without departing from the principle of this application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of this application.

Claims

1. A text processing method based on semantic density, characterized in that The method includes: Obtaining a target text, and performing feature extraction on the target text to obtain multiple text features of each paragraph of the target text; Calculating a semantic density score for each paragraph according to the text features of each paragraph; wherein, calculating a semantic density score for each paragraph according to the text features of each paragraph includes: for each paragraph, calculating a feature value of each text feature of the paragraph by using a distilled language model, and obtaining a target feature value of each text feature by mapping the feature value of each text feature to the interval [0, 1]; and performing weighted fusion calculation on the respective target feature values to obtain the semantic density score of the paragraph; Classifying each paragraph according to the semantic density score of each paragraph to obtain a paragraph category for each paragraph; wherein, the paragraph category is high density or low density; If the paragraph category of the paragraph is high density, performing chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph; If the paragraph category of the paragraph is low density, processing the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph; wherein, the other paragraphs are paragraphs other than the paragraph.

2. The method according to claim 1, characterized in that, The obtaining the target text, and performing feature extraction on the target text to obtain multiple text features of each paragraph of the target text includes; Obtaining a target text to be processed; wherein, the target text includes multiple paragraphs; For each paragraph, inputting the paragraph into a distilled language model, enabling the distilled language model to count the number of entities in the paragraph through named entity recognition, and calculating the entity density of the paragraph according to the number of entities; calculating the information entropy of the paragraph through word frequency distribution; counting the occurrence times of each logical connective in the paragraph, and calculating the logical connection strength of the paragraph according to the occurrence times of each logical connective; identifying multiple core phrases of the paragraph according to a preset core phrase recognition algorithm, and calculating the core phrase coverage rate of the paragraph according to the paragraph and its multiple core phrases; Wherein, the entity density, information entropy, logical connection strength, and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

3. The method according to claim 1, wherein The classifying each paragraph according to the semantic density score of each paragraph to obtain a paragraph category for each paragraph includes: For each paragraph, determining whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold; If the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determining that the paragraph category of the paragraph is high density; If the threshold density score of the paragraph is less than the preset semantic density score threshold, determining that the paragraph category of the paragraph is low density.

4. The method according to claim 1, wherein Performing chunking on the paragraph according to the text type of the target text and the semantic density score of the paragraph to obtain multiple text chunks corresponding to the paragraph, including: Filtering out a target base chunk threshold matching the paragraph from each preset base chunk threshold according to the text type of the target text; Determining a chunking threshold for the paragraph according to the target base chunk threshold and the semantic density score of the paragraph, and performing initial chunking on the paragraph according to the chunking threshold and a preset expansion window to obtain at least one chunking information; wherein, the chunking information includes at least one break point; If the chunking information includes one break point, determining the break point as the target break point of the chunking information; If the chunking information includes multiple break points, obtaining the text content related to each break point from within the paragraph, and calculating a break point score for each break point according to the text content of each break point; Filtering out the target break point of the chunking information from each of the break points according to the break point scores of each of the break points; Chunking the paragraph according to the target break points of each of the chunking information to obtain multiple text chunks corresponding to the paragraph.

5. The method according to claim 1, wherein The processing of the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the paragraphs and their semantic density scores of each other includes: Identifying at least one target other paragraph having a coherence relationship with the paragraph from each of the other paragraphs; Judging whether there is at least one target other paragraph with a low density among each of the target other paragraphs according to the paragraph category of each of the target other paragraphs; If so, judging whether there is a target other paragraph with a low density that meets the merging condition according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of at least one target other paragraph with a low density; If so, merging the paragraph and the target other paragraph with a low density that meets the merging condition to obtain one text chunk; If not, taking the paragraph as one text chunk.

6. A text processing system based on semantic density, characterized in that, The system includes: A feature extraction module, configured to obtain a target text and perform feature extraction on the target text to obtain multiple text features of each paragraph of the target text; A semantic density score calculation module, configured to calculate a semantic density score for each paragraph according to the text features of each paragraph; wherein, the semantic density score calculation module is specifically configured to: for each paragraph, calculate a feature value of each text feature of the paragraph by using a distilled language model, map the feature value of each text feature to the range of [0, 1] to obtain a target feature value of each text feature; and perform weighted fusion calculation on each of the target feature values to obtain the semantic density score of the paragraph; A classification module, configured to classify each paragraph according to the semantic density scores of each paragraph to obtain the paragraph category of each paragraph; wherein, the paragraph category is high density or low density; A high-density paragraph processing module, which is used to, if the paragraph category of the paragraph is high-density, perform chunking processing on the paragraph according to the text type of the target text and the semantic density score of the paragraph, so as to obtain a plurality of text chunks corresponding to the paragraph; A low-density paragraph processing module, which is used to, if the paragraph category of the paragraph is low-density, perform processing on the paragraph according to the text type of the target text, the paragraph and its semantic density score, and the semantic density scores of each other paragraph and its semantic density score; wherein, the other paragraphs are paragraphs other than the paragraph.

7. The system according to claim 6, characterized in that, The feature extraction module includes: A text acquisition unit, which is used to acquire a target text to be processed; wherein, the target text includes a plurality of paragraphs; A feature extraction unit, which is used for each paragraph, input the paragraph into the distilled language model, enable the distilled language model to count the number of entities in the paragraph through named entity recognition, and calculate the entity density of the paragraph according to the number of entities; calculate the information entropy of the paragraph through word frequency distribution; count the occurrence times of each logical connective in the paragraph, and calculate the logical connection strength of the paragraph according to the occurrence times of each logical connective; identify the paragraph according to a preset core phrase recognition algorithm to obtain a plurality of core phrases of the paragraph, and calculate the core phrase coverage rate of the paragraph according to the paragraph and its plurality of core phrases; Wherein, the entity density, information entropy, logical connection strength and core phrase coverage rate of the paragraph constitute multiple text features of the paragraph.

8. The system according to claim 7, characterized in that, The classification module includes: A judgment unit, which is used for each paragraph to judge whether the semantic density score of the paragraph is greater than or equal to a preset semantic density score threshold; A first determination unit, which is used to, if the semantic density score of the paragraph is greater than or equal to the preset semantic density score threshold, determine that the paragraph category of the paragraph is high-density; A second determination unit, which is used to, if the threshold density score of the paragraph is less than the preset semantic density score threshold, determine that the paragraph category of the paragraph is low-density.

Images