Text processing method and device and related equipment

Abstract

The invention provides a text processing method and device and related equipment, and relates to the technical field of text processing. The method comprises the following steps: receiving text information and question information of a user about the text information; dividing the text information into a plurality of sentences, generating a feature vector of each sentence, and calculating the similarity between the feature vectors of any two adjacent sentences in the text information; determining a breakpoint set of the text information based on the similarity between the feature vectors of any two adjacent sentences; performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first fragments; and reply information of the question information is generated based on the plurality of first fragments. Through the technical means, the problem of poor fragment quality in retrieval enhancement generation in related technologies is solved.

Description

Technical Field

[0001] This disclosure relates to the field of text processing technology, and in particular to a text processing method, apparatus and related equipment. Background Technology

[0002] Current retrieval enhancement generation methods face multiple challenges in practical applications. Poor fragmentation quality is a core bottleneck: mainstream methods generally employ fixed-length segmentation or simple syntactic fragmentation based on punctuation, ignoring the inherent semantic structure of documents. This results in the fragmentation of key information, severely impacting the coherence and accuracy of subsequent retrieval and generation. Low-quality fragmentation easily introduces a large amount of irrelevant or redundant content, significantly increasing retrieval noise and interfering with the judgment of the generation model. Summary of the Invention

[0003] This disclosure provides a text processing method, apparatus, and related equipment to improve the quality of fragmentation in retrieval enhancement generation.

[0004] According to one aspect of this disclosure, a text processing method is provided, comprising: receiving text information and user questions about the text information; dividing the text information into multiple sentences, generating a feature vector for each sentence, and calculating the similarity between the feature vectors of any two adjacent sentences in the text information; determining a set of breakpoints in the text information based on the similarity between the feature vectors of any two adjacent sentences; performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first segments; and generating response information to the question based on the multiple first segments.

[0005] In one embodiment of this disclosure, determining a set of breakpoints in text information based on the similarity between the feature vectors of any two adjacent sentences includes: calculating the rate of change of similarity within each window of the text information using a sliding window based on the similarity between the feature vectors of any two adjacent sentences; and determining a set of breakpoints in the text information based on the rate of change of similarity within each window.

[0006] In one embodiment of this disclosure, before performing density clustering on multiple sentences based on the breakpoint set to obtain multiple first segments, the method further includes: treating each sentence as a graph node, using the similarity between any two adjacent sentences as the edge weight of the graph node corresponding to the two adjacent sentences, and constructing a semantic graph corresponding to the text information; and optimizing the breakpoint set using a graph neural network based on the semantic graph.

[0007] In one embodiment of this disclosure, generating response information for a question based on multiple first segments includes: merging and splitting multiple first segments based on segmentation granularity to obtain multiple second segments, wherein the segmentation granularity is the minimum number of tokens and the maximum number of tokens in a segment; determining the target segment corresponding to the question from the multiple second segments; and generating response information for the question based on the target segment.

[0008] In one embodiment of this disclosure, before merging and splitting multiple first segments based on the segmentation granularity to obtain multiple second segments, the method further includes: determining the content complexity of the text information, wherein the content complexity includes syntactic complexity, semantic density, and logical complexity; obtaining real-time resource information of the hardware; and determining the segmentation granularity based on the content complexity and the real-time resource information.

[0009] In one embodiment of this disclosure, determining the fragmentation granularity based on content complexity and real-time resource information includes: determining the fragmentation granularity using a multilayer perceptron based on content complexity; and dynamically adjusting the fragmentation granularity based on real-time resource information.

[0010] In one embodiment of this disclosure, determining the content complexity of text information includes: generating a grammatical dependency tree for each sentence; determining syntactic complexity based on the grammatical dependency tree for each sentence; determining semantic density based on entity frequency and term frequency of the text information; and determining logical complexity based on the number of argument levels of the text information.

[0011] In one embodiment of this disclosure, the method further includes: taking the feature vector of each sentence, the similarity between the feature vectors of any two adjacent sentences, and the real-time resource information of the hardware as states, determining actions through a policy network, wherein the actions include the numerical value of the segmentation granularity and whether to segment; segmenting the text information into multiple third segments based on the actions; and generating response information based on the multiple third segments.

[0012] In one embodiment of this disclosure, after generating response information based on actions, the method further includes: determining the question-answering accuracy of the response information, obtaining user satisfaction with the response information and the resource consumption for generating the response information; determining a reward value based on the question-answering accuracy, satisfaction, and resource consumption; calculating a loss value using a loss function based on the reward value; and optimizing the policy network based on the loss value.

[0013] In one embodiment of this disclosure, after optimizing the policy network based on the loss value, the method further includes: optimizing the window size of the multilayer perceptron and the sliding window using the policy network, wherein the multilayer perceptron is used to determine the granularity of the fragmentation, which is related to the generation of response information, and the sliding window is used to determine the similarity change rate, which is related to the determination of the breakpoint set.

[0014] According to another aspect of this disclosure, a text processing apparatus is provided, comprising: a receiving unit configured to receive text information and user question information regarding the text information; a processing unit configured to divide the text information into multiple sentences, generate a feature vector for each sentence, and calculate the similarity between the feature vectors of any two adjacent sentences in the text information; a determining unit configured to determine a set of breakpoints in the text information based on the similarity between the feature vectors of any two adjacent sentences; a clustering unit configured to perform density clustering on the multiple sentences based on the set of breakpoints to obtain multiple first segments; and a generating unit configured to generate response information to the question information based on the multiple first segments.

[0015] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any of the methods described above by executing the executable instructions.

[0016] According to another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0017] According to another aspect of this disclosure, a computer program product is provided, including computer instructions stored in a computer-readable storage medium, which, when executed by a processor, implement operation instructions for any of the methods described above.

[0018] In the embodiments of this disclosure, a set of breakpoints in the text information is determined based on the similarity between the feature vectors of any two adjacent sentences. Based on the breakpoint set, multiple sentences are subjected to density clustering to obtain multiple first segments. This solves the problem of poor segmentation quality in the generation of related technology retrieval enhancement, thereby improving the segmentation quality in the generation of retrieval enhancement.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 A schematic diagram of a text processing system according to an embodiment of the present disclosure is shown.

[0022] Figure 2A flowchart of a text processing method according to an embodiment of this disclosure is shown.

[0023] Figure 3 A flowchart illustrating a method for dynamically adjusting the granularity of a segmentation according to an embodiment of this disclosure is shown.

[0024] Figure 4 A flowchart illustrating a method for determining content complexity in an embodiment of this disclosure is shown.

[0025] Figure 5 A flowchart of a response information generation method according to an embodiment of this disclosure is shown.

[0026] Figure 6 A schematic diagram of an intelligent question-answering system is shown in an embodiment of this disclosure.

[0027] Figure 7 A schematic diagram of a text processing apparatus according to an embodiment of the present disclosure is shown.

[0028] Figure 8 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0029] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0030] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0031] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0032] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0033] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0034] It should be noted that, unless otherwise specified, the embodiments of this disclosure and the technical features thereof can be combined with each other.

[0035] To facilitate understanding, the following is an explanation of several terms used in this disclosure: Token: A token is the smallest unit for digitally representing text in natural language processing. It can be a character, a word, or a sub-word unit, depending on the word segmentation strategy used. In the model input, the text is converted into a series of token sequences, and each token corresponds to a unique identifier.

[0036] Deep learning models: a class of machine learning models based on artificial neural networks with multi-layer nonlinear transformation structures. They can automatically learn hierarchical feature representations from large amounts of data and are widely used in tasks such as image recognition, speech processing, and natural language understanding.

[0037] RAG (Retrieval-Augmented Generation): A technical framework that combines external knowledge retrieval with language model generation capabilities. This framework first retrieves content fragments that are semantically related to the question information from a given document set, and then generates response information based on these fragments, thereby improving the factual accuracy and contextual relevance of the generated results.

[0038] GNN (Graph Neural Network): A type of neural network specifically designed for processing graph-structured data. It learns representations of nodes or the entire graph by propagating and aggregating information between nodes.

[0039] MLP (Multilayer Perceptron): A type of feedforward artificial neural network consisting of an input layer, one or more hidden layers, and an output layer. All layers are fully connected and complex function mappings are achieved through non-linear activation functions.

[0040] PPO (Proximal Policy Optimization) algorithm: A policy gradient method for reinforcement learning that introduces a pruning mechanism to limit the policy update magnitude, thereby achieving efficient policy optimization while ensuring training stability.

[0041] Transformer model: A deep learning model based on self-attention mechanism and feedforward neural network. This model abandons the traditional loop structure and relies entirely on attention mechanism to capture the dependency relationship between any position in the sequence. It is widely used in natural language processing tasks.

[0042] LLM-as-a-judge evaluation (Large Language Model as a Judge): A technique that uses a large language model to automatically score, compare, or judge generated content. This technique guides the large language model to output evaluation results based on preset criteria (such as accuracy, relevance, and helpfulness) by designing prompt templates.

[0043] Spacy is an open-source, industrial-grade natural language processing library that supports functions such as sentence segmentation, part-of-speech tagging, named entity recognition, and dependency parsing.

[0044] SimCSE (Simple Contrastive Learning of Sentence Embeddings) is a deep learning model that generates high-quality sentence semantic vectors through unsupervised or supervised contrastive learning.

[0045] Dropout Augmentation: A technique that applies different Dropout masks multiple times to the same input during the model inference phase to generate diverse semantic representations; in the SimCSE model, it is used to construct positive sample pairs to improve the robustness and discriminative ability of sentence vectors.

[0046] HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm: A density clustering method that can automatically identify noise points and discover clusters of arbitrary shapes without requiring a preset number of clusters.

[0047] Retrieval-Augmented Generation: A technical framework that combines external knowledge retrieval with language model generation capabilities. It first retrieves content related to the question from a large-scale document library, and then generates an answer based on the retrieval results.

[0048] GPU (Graphics Processing Unit): A graphics processor is a hardware device that excels at parallel computing and is widely used to accelerate the training and inference of deep learning models.

[0049] Figure 1 This illustration shows a schematic diagram of a text processing system according to an embodiment of the present disclosure. The text processing system includes: Server 101 and client 102.

[0050] The server 101 may be equipped with an application that performs the following actions: receiving text information and user questions about the text information; dividing the text information into multiple sentences, generating a feature vector for each sentence, and calculating the similarity between the feature vectors of any two adjacent sentences in the text information; determining a set of breakpoints in the text information based on the similarity between the feature vectors of any two adjacent sentences; performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first segments; and generating response information to the question based on the multiple first segments.

[0051] Figure 2 This diagram illustrates a flowchart of a text processing method according to an embodiment of the present disclosure. This method can be applied to various servers. The method is as follows: Figure 2 As shown, it includes the following steps: S201, Receive text information and user questions about the text information.

[0052] For example, textual information refers to the content of a document to be processed that is entered by the user, including but not limited to natural language text such as technical documents, explanatory texts, or academic papers.

[0053] As an example, a question refers to a query or question posed by a user in response to textual information, which guides the generation of subsequent response information.

[0054] S202: Divide the text information into multiple sentences, generate a feature vector for each sentence, and calculate the similarity between the feature vectors of any two adjacent sentences in the text information.

[0055] As an example, a method combining rule-based regular expression matching with deep learning models (such as Spacy's sentence segmenter) can be used to ensure that complex sentence structures (such as compound sentences with semicolons) are accurately segmented.

[0056] As an example, the SimCSE model is used to extract the feature vector for each sentence.

[0057] In one exemplary embodiment, the SimCSE model is trained using a contrastive learning loss function, as shown in Equation (1): (1) in, Let i be the feature vector of the i-th sentence. Let be the feature vector of the positive sample of the i-th sentence. Let be the feature vector of the negative sample of the i-th sentence. Let be the temperature coefficient, and let be the total number of negative samples for the i-th sentence. Let sim() be the cosine similarity function. The i-th sentence, after being augmented by Dropout, is used as its positive sample, and the other sentences are used as negative samples for the i-th sentence.

[0058] In an exemplary embodiment, the vector of the i-th sentence is calculated according to formula (2). With the vector of the (i+1)th sentence Cosine similarity: (2) Where || is used to find the 2-norm.

[0059] Construct the semantic similarity matrix M This is used for subsequent breakpoint detection.

[0060] S203, based on the similarity between the feature vectors of any two adjacent sentences, determine the set of breakpoints in the text information.

[0061] An exemplary breakpoint set refers to a set of candidate segmentation points determined based on locations where the similarity between adjacent sentences changes significantly, used to identify potential semantic boundaries in the text.

[0062] S204, based on the breakpoint set, performs density clustering on multiple sentences to obtain multiple first segments.

[0063] As an example, the HDBSCAN algorithm can be used for density clustering, which aggregates consecutive sentences between breakpoints into semantically coherent units according to their semantic tightness.

[0064] As an example, each first segment contains several semantically related sentences.

[0065] S205, generate response information based on multiple first fragments of the question information.

[0066] In a demonstrative sense, response information refers to the answer output by using retrieval enhancement generation technology, which combines the content of multiple first fragments with the question information.

[0067] In this embodiment, the text information is divided into multiple sentences, and a feature vector is generated for each sentence. Next, the similarity between the feature vectors of any two adjacent sentences is calculated, and a set of breakpoints is determined based on the locations of abrupt changes in similarity. Then, based on this set of breakpoints, the sentence sequence is density-clustered to form multiple first segments. Finally, these first segments are used to generate response information for the query. The entire process focuses on semantic coherence, avoiding information fragmentation caused by fixed-length segmentation. Through these technical means, the semantic integrity of the segments is effectively improved, thereby reducing retrieval noise and improving the accuracy of the response information.

[0068] In one embodiment of this disclosure, determining a set of breakpoints in text information based on the similarity between the feature vectors of any two adjacent sentences includes: calculating the rate of change of similarity within each window of the text information using a sliding window based on the similarity between the feature vectors of any two adjacent sentences; and determining a set of breakpoints in the text information based on the rate of change of similarity within each window.

[0069] As an example, a sliding window is a continuous subsequence window that moves at fixed or dynamic steps over a sequence of sentences in textual information, used for local observation of semantic changes.

[0070] As an example, the similarity change rate refers to the degree of fluctuation in the similarity between adjacent sentences within a sliding window. It is quantified by methods such as the standard deviation, variance, or maximum decrease in similarity within the statistical window and is used to identify semantically abrupt regions.

[0071] In this embodiment, a sliding window is applied to the sentence sequence of text information, and the rate of change of similarity between the feature vectors of adjacent sentences within each window is calculated. When the rate of change of similarity within a certain window exceeds a preset threshold, the last position of that window is marked as a candidate breakpoint. Finally, all candidate breakpoints are summarized to form a breakpoint set, which serves as the boundary basis for subsequent density clustering. Through the above technical means, breakpoint detection can focus on local semantic mutation regions, avoiding boundary blurring caused by global averaging effects, thereby improving the semantic cohesion of the first segment.

[0072] Sliding window detection: Define the window size w (default value is 3 lines), and calculate the average similarity change rate within the window: In an exemplary embodiment, if ,in, The rate of change of similarity within a window. If the threshold is set to 0.3, the last position of the window will be marked as a candidate breakpoint.

[0073] Window definition: The sliding window has a default size of "3 consecutive sentences" (w=3), that is, each window covers the i-th sentence, the (i+1)-th sentence, and the (i+2)-th sentence (i is the starting sentence number of the window); Core calculation: For the current window, calculate the "average similarity change rate Δsimi" - this indicator reflects the degree of abrupt change in semantic coherence between sentences within the window. In essence, it measures the risk of semantic connection break by the difference in semantic similarity between adjacent sentences (cosine similarity calculated based on sentence vectors generated by SimCSE). Breakpoint marking rules: when When the threshold is 0.3, it indicates that there is a significant change in the semantic coherence within the current window. At this time, the boundary between the last sentence (i.e., the i+2nd sentence) and the next sentence (i+3rd sentence) of the current window is marked as a candidate breakpoint, that is, the segmentation position of the subsequent segments is set at the end of the i+2nd sentence.

[0074] In short, the core of sliding window detection is to capture semantic mutations through "3-sentence windows". The candidate breakpoints eventually fall at the junction of the current window's ending sentence and the next sentence, ensuring that the segmentation does not sever the semantics of a single sentence, while accurately locating the breakpoints of semantic logic.

[0075] In one embodiment of this disclosure, before performing density clustering on multiple sentences based on the breakpoint set to obtain multiple first segments, the method further includes: treating each sentence as a graph node, using the similarity between any two adjacent sentences as the edge weight of the graph node corresponding to the two adjacent sentences, and constructing a semantic graph corresponding to the text information; and optimizing the breakpoint set using a graph neural network based on the semantic graph.

[0076] As an example, a semantic graph is a graph structure constructed with each sentence as a node and the similarity between any two adjacent sentences as the weight of the corresponding edge, used to represent the semantic relationships between sentences in textual information.

[0077] In this embodiment, each sentence in the text is modeled as a node in a semantic graph, with the similarity between adjacent sentences serving as edge weights. This semantic graph is then input into a graph neural network, which uses its context-aware capabilities to refine each candidate breakpoint location. For example, it suppresses false breakpoints caused by local noise or enhances true breakpoints at logical transitions between sentences. The optimized breakpoint set is then used in subsequent density clustering steps. Through these techniques, the accuracy of the breakpoint set is improved, making the boundaries upon which density clustering relies more closely match the deep semantic structure of the text, thereby enhancing the logical integrity of the first segment.

[0078] In an exemplary embodiment, the semantic graph G=(V,E). The loss function used when training the GNN is Equation (4): (4) in, Indicate whether it is a real breakpoint. For predicting probabilities, v is the index of the breakpoint.

[0079] The specific process logic is as follows: all sentences are treated as graph nodes V, and the semantic similarity between sentences (cosine similarity calculated based on SimCSE vectors) is used as the edge weight E to construct a semantic graph G=(V,E); then, the nodes are classified by GCN (to determine whether the node corresponds to a real breakpoint, with the label y_v∈{0,1}), and the model is optimized with the cross-entropy loss function. Finally, the "real breakpoint" is selected from the candidate breakpoints in the sliding window, thus achieving "refinement" of breakpoint determination.

[0080] Sliding window detection is positioned as "preliminary candidate screening," relying solely on the similarity change rate within a local window without considering the global semantic associations of the entire document. This may lead to misjudgments (such as false positive candidate breakpoints caused by local semantic fluctuations in short text fragments). GNN refinement, on the other hand, is positioned as "global precise optimization." It incorporates global associations between sentences into the judgment logic through semantic graph construction and then achieves precise breakpoint identification through GCN node classification (combined with supervised optimization of the cross-entropy loss function). Its results are more in line with the core goal of "semantic coherence" (i.e., the requirement in the technical briefing to "improve the relevance and completeness of search content"). The core function of GNN refinement is to correct the coarse screening results of sliding window detection. Therefore, when the results of the two conflict, the breakpoint results after GNN refinement shall prevail to ensure that the final generated semantic segmentation logic is more coherent and accurate.

[0081] In an exemplary embodiment, the HDBSCAN algorithm is used for density clustering, and its objective function is shown in Equation (5): (5) in: Let len(c) be the maximum distance within cluster c, len(c) be the number of tokens in cluster c, and C be the number of clusters. The minimum distance between clusters This is the balance coefficient.

[0082] In one embodiment of this disclosure, generating response information for a question based on multiple first segments includes: merging and splitting multiple first segments based on segmentation granularity to obtain multiple second segments, wherein the segmentation granularity is the minimum number of tokens and the maximum number of tokens in a segment; determining the target segment corresponding to the question from the multiple second segments; and generating response information for the question based on the target segment.

[0083] As an example, the fragment granularity is used to constrain the length of the fragment ultimately used for retrieval.

[0084] As an example, the second segment refers to the final retrieval unit that meets the token number constraint, obtained by merging or splitting the first segment according to the granularity of the segmentation.

[0085] As an example, the target fragment refers to the fragment most relevant to the question information selected from multiple second fragments through semantic matching or relevance scoring, which serves as the basis for generating the response information.

[0086] In this embodiment, after obtaining multiple first segments, these segments are adjusted according to the segmentation granularity: if the length of a first segment is less than the minimum number of tokens, it is attempted to merge with its adjacent segments; if it exceeds the maximum number of tokens, it is split according to semantic boundaries, thereby generating multiple second segments that meet the length constraints; subsequently, the target segment most semantically relevant to the question information is retrieved from these second segments, and the final response information is generated based on the content of the target segment. Through the above technical means, the segment used to generate the final response information satisfies the model input length limit while maintaining sufficient semantic context, thereby improving the relevance and coherence of the response information.

[0087] In an exemplary embodiment, for segments whose length is less than the minimum number of tokens, their semantic consistency with the preceding / following segments is calculated according to formula (6): (6) in, , and These are the previous, current, and next fragments, respectively. If `merge_score` > 0.7, the merge operation is performed. Simultaneously, protection flags are set for fragments containing high-frequency entities (such as technical terms) to prevent accidental merging.

[0088] In one embodiment of this disclosure, before merging and splitting multiple first segments based on the segmentation granularity to obtain multiple second segments, the method further includes: determining the content complexity of the text information, wherein the content complexity includes syntactic complexity, semantic density, and logical complexity; obtaining real-time resource information of the hardware; and determining the segmentation granularity based on the content complexity and the real-time resource information.

[0089] As an example, content complexity refers to a comprehensive indicator used to measure the internal structure and information density of textual information, reflecting the difficulty of processing the text at the syntactic, semantic, and logical levels.

[0090] Exemplary syntactic complexity refers to the average dependency tree depth calculated based on the grammatical dependency tree of each sentence, used to characterize the nesting and complexity of sentence structure.

[0091] As an example, semantic density refers to the frequency of occurrence of named entities and domain terms per unit length, including entity frequency and term frequency, and is used to measure the density of key information in a text.

[0092] As an example, logical complexity refers to the number of logical levels, such as arguments and evidence, in a text, reflecting the depth of the document's reasoning structure.

[0093] Exemplary, real-time resource information refers to hardware operating status data at the current moment, including metrics such as GPU memory utilization.

[0094] Figure 3 This diagram illustrates a flowchart of a method for dynamically adjusting the granularity of fragmentation according to an embodiment of the present disclosure. The method is as follows: Figure 3 As shown, it includes the following steps: S301, based on content complexity, uses a multilayer perceptron to determine the granularity of the slice; S302 dynamically adjusts the granularity of fragmentation based on real-time resource information.

[0095] In this embodiment, an initial fragmentation granularity is first predicted using a multilayer perceptron based on content complexity. Then, this initial fragmentation granularity is dynamically adjusted in conjunction with real-time hardware resource information. For example, when GPU memory utilization is high, the maximum number of tokens is reduced to decrease memory usage, or the length limit is appropriately relaxed when resources are abundant to preserve a more complete semantic context. Through these techniques, the fragmentation granularity can respond to system runtime resource constraints while taking into account the semantic characteristics of the text, achieving a synergistic optimization of semantic integrity and resource efficiency.

[0096] Figure 4 This diagram illustrates a flowchart of a content complexity determination method according to an embodiment of the present disclosure. The method is as follows: Figure 4 As shown, it includes the following steps: S401, generates the grammar dependency tree for each sentence; S402 determines the syntactic complexity based on the grammatical dependency tree of each sentence; S403, determine semantic density based on entity frequency and term frequency of text information; S404 determines the logical complexity based on the number of argument levels in the textual information.

[0097] An exemplary grammatical dependency tree is a tree structure built for each sentence by a dependency parser.

[0098] In one exemplary embodiment, dependency parsing is performed on each sentence to extract the core subject-verb-object structure and modification relations, and a grammatical dependency tree is constructed.

[0099] Mathematical expression example: Let the sentence be The dependency relation set is obtained through dependency parsing: }; in, Indicator and Grammatical relationships (such as subject-verb, verb-object).

[0100] As an example, entity frequency refers to the number of times named entities (such as people's names, organization names, and technical component names) appear per unit length in textual information, and is used to quantify the density of specific objects in the text.

[0101] As an example, term frequency refers to the number of times domain keywords or technical terms (such as "self-attention" or "backpropagation" in the field of machine learning) appear per unit length in textual information, and is used to quantify the density of abstract concepts in the text.

[0102] As an example, the number of argument levels refers to the nesting depth of the argument structure in the textual information, such as the number of layers of sub-arguments under the main argument, and the number of supporting evidence under the sub-arguments, which is used to characterize the complexity of logical reasoning.

[0103] In this embodiment, syntactic complexity is determined by generating a grammatical dependency tree for each sentence and calculating its average dependency tree depth; semantic density is determined by statistically analyzing entity and term frequencies; and logical complexity is determined by analyzing the argument structure of the text to obtain the number of argument levels. These three factors together constitute a complete representation of content complexity. Through these technical means, multi-dimensional and fine-grained semantic difficulty criteria are provided for dynamic decision-making at the segmentation granularity level, enabling the segmentation strategy to accurately adapt to different types of text content.

[0104] In an exemplary embodiment, the syntactic complexity is calculated according to formula (7): Statistical average grammatical dependency tree depth: (7) in For the first The grammatical dependency tree for each sentence has a total of n sentences.

[0105] In one exemplary embodiment, the entity frequency is calculated according to formula (8): (8) In an exemplary embodiment, the logical complexity is calculated according to formula (9): (9) Similar to the method used to calculate entity frequencies, the term frequency is calculated and denoted as ft.

[0106] In one exemplary embodiment, a lightweight MLP model (i.e., a multilayer perceptron) is constructed, with the input feature vector... Output window size The model structure includes a first hidden layer, a second hidden layer, and an output layer.

[0107] Calculation of the first hidden layer h1: h1=ReLU(W1x+b1); (10) Where W1 is the weight matrix of the first hidden layer, b1 is the bias, and ReLU is the activation function to achieve nonlinear transformation.

[0108] Calculation of the second hidden layer h2: h2=ReLU(W2h1+b2); (11) Where W2 is the weight matrix of the second hidden layer, b2 is the bias, and features are further extracted based on h1.

[0109] Calculation of output layer w: w=ReLU(W3h2+b3);(12) Where W3 is the output layer weight matrix, b3 is the bias, the final window size is output based on the features of h2, and subsequent processing is required to ensure that w falls within the set {64,128,256,512}.

[0110] Construct a lightweight MLP (Multilayer Perceptron) model. The input feature vector (composed of "average dependency tree depth, entity frequency, and number of logical levels") outputs a window size appropriate to the content complexity, with selectable values ​​of {64, 128, 256, 512} tokens. For example, academic papers (high complexity) might output a small window of 64 / 128 tokens, while general explanatory texts (low complexity) might output a large window of 256 / 512 tokens, ensuring the window granularity matches the content.

[0111] In one exemplary embodiment, GPU memory utilization is monitored in real time. The maximum window threshold is dynamically adjusted according to formula (13): (13) To prevent the "content-fitted window" from exceeding the hardware resource capacity, the maximum window threshold is dynamically compressed by monitoring GPU memory utilization in real time to ensure stable system operation. If GPU memory is sufficient, a larger window can be retained (ensuring fragment integrity); if memory is scarce, the window is compressed (reducing resource consumption) to avoid system lag or memory overflow. For example, when GPU memory utilization reaches 95% during peak periods, even if the content complexity is low, the window needs to be compressed to prioritize system responsiveness. The maximum window threshold is dynamically adjusted through "real-time GPU memory utilization," meaning the maximum window threshold is not a fixed value but changes dynamically with the hardware resource status, ranging from a maximum of 512 tokens to a minimum of 128 tokens.

[0112] Figure 5 This diagram illustrates a flowchart of a response information generation method according to an embodiment of the present disclosure. The method is as follows: Figure 5 As shown, it includes the following steps: S501 uses the feature vector of each sentence, the similarity between the feature vectors of any two adjacent sentences, and the real-time resource information of the hardware as the state, and determines the action through the policy network. The action includes the numerical value of the segmentation granularity and whether to segment. S502, based on actions, divide text information into multiple third segments; S503 generates response information based on multiple third fragments.

[0113] As an example, state refers to the environmental observation information that the policy network relies on when making decisions, including the feature vector of each sentence, the similarity between the feature vectors of any two adjacent sentences, and real-time hardware resource information.

[0114] An exemplary policy network is a decision-making model built on reinforcement learning, used to output actions based on the current state.

[0115] As an example, the third segment refers to the segment obtained by directly segmenting text information according to the action output by the policy network. Its length and boundaries are jointly determined by the segmentation granularity and segmentation decision in the action.

[0116] In this embodiment, the feature vector of each sentence, the similarity between adjacent sentences, and real-time hardware resource information are integrated into a state input policy network. Based on this, the policy network outputs an action indicating whether to perform segmentation and what segmentation granularity to use. Subsequently, the text information is segmented into multiple third segments according to this action, and response information is generated based on these third segments to address the question. Through these technical means, the segmentation behavior can be dynamically controlled by an end-to-end learnable policy, achieving joint optimization of semantic structure and system resources, thereby improving the overall response quality and adaptability of the question-answering system.

[0117] In one embodiment of this disclosure, after generating response information based on actions, the method further includes: determining the question-answering accuracy of the response information, obtaining user satisfaction with the response information and the resource consumption for generating the response information; determining a reward value based on the question-answering accuracy, satisfaction, and resource consumption; calculating a loss value using a loss function based on the reward value; and optimizing the policy network based on the loss value.

[0118] For example, question-and-answer accuracy refers to the degree of semantic and content consistency between the generated response information and the standard answer or factual basis, and is used to measure the correctness of the system's response.

[0119] Exemplary satisfaction refers to users' subjective evaluation feedback on response information, including explicit ratings or implicit behavioral signals, used to reflect the quality of user experience.

[0120] For example, resource consumption refers to the computing resources used in generating response information, including GPU memory increments, processing latency, or total token processing volume.

[0121] As an example, the reward value refers to a scalar feedback signal calculated based on question-answering accuracy, satisfaction, and resource consumption, which is used to guide the learning direction of the policy network.

[0122] As an example, the loss function refers to the objective function used to train the policy network, which constructs an advantage function based on the reward value.

[0123] As an example, the loss value refers to the value calculated by substituting the output of the current policy network under a given state-action pair into the loss function, and is used for backpropagation to update the network parameters.

[0124] In this embodiment, after generating the response information, the accuracy of the question-and-answer session is evaluated, user satisfaction with the response information is obtained, and resource consumption during the generation process is recorded. Then, a reward value is calculated based on these three indicators. This reward value is then used to construct an advantage function and substituted into a loss function to obtain the loss value. Finally, optimization algorithms such as gradient descent are used to update the parameters of the policy network based on this loss value, achieving online evolution of the policy. Through these technical means, the policy network can continuously learn from real-world interactive feedback, gradually optimizing segmented decision-making and balancing response quality, user experience, and system efficiency.

[0125] In one embodiment of this disclosure, after optimizing the policy network based on the loss value, the method further includes: optimizing the window size of the multilayer perceptron and the sliding window using the policy network, wherein the multilayer perceptron is used to determine the granularity of the fragmentation, which is related to the generation of response information, and the sliding window is used to determine the similarity change rate, which is related to the determination of the breakpoint set.

[0126] As an example, window size refers to the number of sentences covered by the sliding window in the sentence sequence of text information. This parameter affects the local observation range of the similarity change rate, and thus affects the sensitivity and accuracy of the breakpoint set.

[0127] In this embodiment, after optimizing the policy network based on the loss value, the piecewise granularity decision or internal representation output by the policy network is further utilized to guide or fine-tune the prediction target of the multilayer perceptron, making its output piecewise granularity closer to the granularity validated by the policy network in real feedback. Simultaneously, the action selected by the policy network based on the current text complexity and resource status can be back-mapped to a suitable window size, adjusting the observation scale when calculating the similarity change rate in subsequent processing. Through these techniques, the piecewise granularity prediction and sliding window size of the multilayer perceptron can benefit from the global optimization target of reinforcement learning, thereby improving the accuracy of breakpoint detection and the adaptability of piecewise granularity, ultimately enhancing the semantic consistency and resource efficiency of the entire question-answering process.

[0128] In one exemplary embodiment, the state space for reinforcement learning is set up as follows: (14) : Current semantic vector (a vector representation of the current sentence or segment generated based on models such as SimCSE, used to characterize the semantic features of the text).

[0129] Similarity value (the semantic similarity between the current sentence or paragraph and other related sentences, paragraphs or questions, reflecting the degree of content relevance).

[0130] GPU memory utilization (real-time monitored hardware resource status, used to balance the resource consumption of the sharding strategy).

[0131] Action space:

[0132] {Segment, Do Not Segment}: Decides whether to segment the current text segment.

[0133] {w1,w2,…,wk}: An optional set of window sizes (e.g., 64, 128, 256, 512 tokens, etc.) used to determine the granularity of the fragmentation.

[0134] Set the reward function: (15) Weighting coefficient for improving question-and-answer accuracy (controls the importance of accuracy in rewards; parameters need to be adjusted according to business needs).

[0135] Accuracy_gain: The improvement in question-answering accuracy after adopting the current sharding strategy (e.g., the difference in accuracy compared to the baseline strategy).

[0136] β: Weighting coefficient of user ratings (controls the proportion of subjective user evaluation in the reward).

[0137] User_score: User's satisfaction rating of the segmented question and answer results (e.g., on a 1-5 scale).

[0138] γ: Weighting coefficient of resource consumption (controls the degree of penalty for resource costs in rewards).

[0139] Resource_cost: Resource consumption (such as GPU memory usage, computation time, etc.) using the current partitioning strategy.

[0140] Setting the loss function during offline pre-training: (16) Offline pre-training: The policy network is pre-trained on historical data or in a simulated environment to enable it to make preliminary decisions, laying the foundation for faster convergence during online training.

[0141] The current policy network (with parameter θ) is in state θ. Select action The probability of.

[0142] The old policy network (parameter θold) in state Select action The probability is used to constrain the magnitude of policy updates.

[0143] At: Advantage Function, representing the action. In state The relative value under the condition (i.e., the difference between the reward of the action and the state baseline reward).

[0144] The clipping factor (usually 0.1 or 0.2) is used to limit the amount of material removed. The scope should be limited to avoid excessively large policy updates.

[0145] clip(( ),1- ,1+ ): Trim operation, restricting the input value to [1- ,1+ Within the specified interval, ensure the smoothness of strategy updates.

[0146] By maximizing this objective function and iteratively updating the policy network parameters θ, the RL agent learns a better partitioning decision policy.

[0147] In one application example, a user asks, "How is the self-attention mechanism of the Transformer model calculated?" and uploads a 50-page AI paper as text information.

[0148] First, the text information is divided into multiple sentences, and a syntactic dependency tree is generated for each sentence. The average dependency tree depth is calculated based on the syntactic dependency tree to obtain the syntactic complexity. Simultaneously, entity frequency and term frequency in the text information are statistically analyzed to determine the semantic density. The argument-evidence structure is analyzed to obtain the number of argument levels, determining the logical complexity. These three factors together constitute the content complexity. GPU memory utilization is simultaneously acquired as real-time resource information.

[0149] Subsequently, each sentence is input into the SimCSE model to generate a feature vector for each sentence; the similarity between the feature vectors of any two adjacent sentences is calculated; a sliding window is applied to the sentence sequence, with the window size determined by the policy network or the default configuration, and the rate of change of similarity within each window is calculated; based on the positions where the rate of change of similarity exceeds a threshold, an initial set of breakpoints is determined.

[0150] Next, each sentence is treated as a graph node, and the similarity between any two adjacent sentences is used as the edge weight of the corresponding graph node to construct a semantic graph. A graph neural network is then used to perform message passing and node representation updates on the semantic graph, classify or re-score candidate breakpoints in the initial breakpoint set, and output the optimized breakpoint set.

[0151] Based on the optimized set of breakpoints, the HDBSCAN algorithm is used to perform density clustering on the continuous sentences between breakpoints to obtain multiple first segments. Among them, the chapters such as "Model Architecture", "Mathematical Derivation" and "Experimental Analysis" are naturally separated into independent first segments due to their close internal semantics and large semantic jumps across chapters. In the "Mathematical Derivation" chapter, the formula block and its adjacent explanatory text are clustered into the same cluster due to their high semantic correlation, forming a sub-fragment structure.

[0152] Further determine the fragment granularity: Based on content complexity, use a multilayer perceptron to predict the initial fragment granularity (e.g., minimum number of tokens is 64, maximum number of tokens is 256); then dynamically adjust the fragment granularity based on GPU memory utilization (e.g., reduce the maximum number of tokens to 192 due to memory pressure). Based on the adjusted fragment granularity, merge and split the first fragment: short fragments with a length of less than 64 tokens are attempted to be merged with adjacent fragments, and long fragments with a length of more than 192 tokens are split according to internal semantic boundaries, ultimately resulting in multiple second fragments.

[0153] The semantic relevance of multiple second segments to the question "How is the self-attention mechanism of the Transformer model calculated?" is calculated, and the target segment—the second segment containing the self-attention formula and its corresponding textual explanation—is selected. Based on this target segment, a response is generated, including the calculation process of the self-attention mechanism, the meaning of each matrix, and a formula example.

[0154] In addition, the accuracy of the generated question-and-answer session (evaluated using LLM-as-a-judge), user satisfaction (user feedback of clicking "helpful"), and resource consumption (peak GPU memory usage) are recorded. A reward value is calculated based on these three metrics. The loss value is calculated using the loss function of the PPO algorithm, and the policy network is optimized accordingly. The optimized policy network is further used to fine-tune the slice-level prediction target of the multilayer perceptron and provides feedback to adjust the sliding window size, offering better parameters for subsequent processing. Through these techniques, this embodiment achieves a 40% improvement in answer accuracy and a 35% reduction in GPU memory usage compared to traditional RAG.

[0155] In one application example, during peak hours on an online education platform, users upload text information and submit questions, at which point GPU memory utilization reaches 95%. First, this real-time resource information is acquired, and the content complexity of the text information (including syntactic complexity calculated based on the dependency tree, semantic density determined based on entity and term frequencies, and logical complexity determined based on the number of argument levels) is used as input. A multilayer perceptron is used to predict the initial fragment granularity based on content complexity. Then, based on the real-time resource information with high memory utilization, the window is dynamically compressed, adjusting the maximum number of tokens for the fragment granularity from 512 to 256, and the minimum number of tokens is correspondingly reduced to maintain semantic integrity. During the fragment generation stage, breakpoints are detected using a sliding window, a semantic graph is constructed, and breakpoints are refined using a graph neural network. The HDBSCAN algorithm is then used to cluster the fragments to obtain the first segment. Subsequently, the first segment is merged and split according to the adjusted fragment granularity to generate the second segment. During this process, fragments with high entity density (i.e., fragments containing "core knowledge points") are prioritized for retention, while low-density fragments are merged or discarded.

[0156] Target fragments relevant to the question are selected from the high-entity second fragment to avoid loading redundant content when generating the response. Simultaneously, question-and-answer accuracy, user satisfaction, and resource consumption are recorded, reward values ​​are calculated, and the policy network is optimized using the PPO algorithm. The optimized policy network further guides the parameter adjustment of the multilayer perceptron and sliding window to adapt to high-load scenarios. Through these technical means, response time is reduced by 50% while ensuring coverage of key knowledge points.

[0157] Figure 6 This diagram illustrates a schematic of an intelligent question-answering system according to an embodiment of the present disclosure. The system is as follows: Figure 6 As shown, the process includes the following: The set of text to be processed by the user and the question information.

[0158] First, document preprocessing and structure extraction are performed, including format parsing, sentence segmentation, and dependency parsing. Then, the SimCSE model is used to generate sentence feature vectors and calculate the semantic similarity between adjacent sentences, constructing a semantic similarity matrix. Based on this, content complexity is evaluated from three dimensions: syntactic complexity (average dependency tree depth), semantic density (entity frequency and term frequency), and logical complexity (number of argument levels). In the fragment generation stage, a sliding window is used to detect semantic abrupt changes to obtain candidate breakpoints. A graph neural network is then used to refine the breakpoints on the semantic graph. HDBSCAN is then used to perform density clustering of sentences between breakpoints to form the first fragment. Short fragment merging and high-frequency entity protection strategies are implemented to output a semantically coherent and length-controllable fragment set.

[0159] On the retrieval enhancement side, the system performs semantic matching retrieval based on these segments and generates response information about the question. At the same time, a reinforcement learning agent is introduced: its state includes sentence feature vectors, similarity between adjacent sentences, and real-time hardware resource information; its actions include whether to segment and the segment granularity (e.g., 64 / 128 / 256 / 512); and the reward is jointly determined by question-answering accuracy, user satisfaction, and resource consumption. After optimization by the PPO algorithm, the policy network is not only used to directly generate the third segment, but also to guide the multilayer perceptron's segment granularity prediction and sliding window size adjustment.

[0160] Ultimately, user feedback drives reward calculation and strategy updates, forming a continuous evolutionary closed loop of "sharding—retrieval—generation—feedback—optimization," thereby achieving a synergistic improvement in resource efficiency and question-answering quality while ensuring semantic integrity.

[0161] Based on the same inventive concept, this disclosure also provides a text processing apparatus, as shown in the following embodiments. Since the principle by which the text processing apparatus solves the problem is similar to that of the above method embodiments, the implementation of the text processing apparatus can refer to the implementation of the above method embodiments, and repeated details will not be elaborated further.

[0162] Figure 7 This disclosure illustrates a text processing apparatus, such as... Figure 7 As shown, the text processing device may include: The receiving unit 701 is configured to receive text information and user questions about the text information. The processing unit 702 is configured to divide the text information into multiple sentences, generate a feature vector for each sentence, and calculate the similarity between the feature vectors of any two adjacent sentences in the text information. The determining unit 703 is configured to determine the set of breakpoints in the text information based on the similarity between the feature vectors of any two adjacent sentences. Clustering unit 704 is configured to perform density clustering on multiple sentences based on a set of breakpoints to obtain multiple first segments; The generation unit 705 is configured to generate response information based on multiple first fragments of the question information.

[0163] In some embodiments, the determining unit 703 is further configured to calculate the rate of change of similarity within each window of the text information by means of a sliding window based on the similarity between the feature vectors of any two adjacent sentences; and to determine the set of breakpoints of the text information based on the rate of change of similarity within each window.

[0164] In some embodiments, the determining unit 703 is further configured to treat each sentence as a graph node, use the similarity between any two adjacent sentences as the edge weight of the graph node corresponding to the two adjacent sentences, construct a semantic graph corresponding to the text information, and optimize the set of breakpoints using a graph neural network based on the semantic graph.

[0165] In some embodiments, clustering unit 704 is further configured to merge and split multiple first segments based on sharding granularity to obtain multiple second segments, wherein the sharding granularity is the minimum number of tokens and the maximum number of tokens in a segment; determine the target segment corresponding to the question information from the multiple second segments; and generate response information to the question information based on the target segment.

[0166] In some embodiments, clustering unit 704 is further configured to determine the content complexity of text information, wherein the content complexity includes syntactic complexity, semantic density, and logical complexity; obtain real-time resource information of the hardware; and determine the granularity of the fragmentation based on the content complexity and the real-time resource information.

[0167] In some embodiments, clustering unit 704 is further configured to determine the granularity of sharding based on content complexity using a multilayer perceptron; and to dynamically adjust the granularity of sharding based on real-time resource information.

[0168] In some embodiments, clustering unit 704 is further configured to generate a syntactic dependency tree for each sentence; determine syntactic complexity based on the syntactic dependency tree for each sentence; determine semantic density based on entity frequency and term frequency of text information; and determine logical complexity based on the number of argument levels of text information.

[0169] In some embodiments, the generation unit 705 is further configured to use the feature vector of each sentence, the similarity between the feature vectors of any two adjacent sentences, and the real-time resource information of the hardware as states, and determine actions through a policy network, wherein the actions include the numerical value of the segmentation granularity and whether to segment; segment the text information into multiple third segments based on the actions; and generate response information based on the multiple third segments.

[0170] In some embodiments, the generation unit 705 is further configured to determine the question-answering accuracy of the response information, obtain the user's satisfaction with the response information and the resource consumption for generating the response information; determine a reward value based on the question-answering accuracy, satisfaction, and resource consumption; calculate a loss value using a loss function based on the reward value; and optimize the policy network based on the loss value.

[0171] In some embodiments, the generation unit 705 is further configured to optimize the window size of the multilayer perceptron and the sliding window using a policy network, wherein the multilayer perceptron is used to determine the fragmentation granularity, which is related to the generation of response information, and the sliding window is used to determine the similarity change rate, which is related to the determination of the breakpoint set.

[0172] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0173] The following reference Figure 8 To describe an electronic device 800 according to such an embodiment of the present disclosure. Figure 8 The electronic device 800 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0174] like Figure 8 As shown, the electronic device 800 is presented in the form of a general-purpose computing device. The components of the electronic device 800 may include, but are not limited to: at least one processor 810, at least one memory 820, and a bus 830 connecting different system components (including memory 820 and processor 810).

[0175] The memory stores program code that can be executed by the processor 810, causing the processor 810 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processor 810 can perform the following steps of the above method embodiments: receiving text information and user questions about the text information; dividing the text information into multiple sentences, generating a feature vector for each sentence, and calculating the similarity between the feature vectors of any two adjacent sentences in the text information; determining a set of breakpoints in the text information based on the similarity between the feature vectors of any two adjacent sentences; performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first segments; and generating response information to the question based on the multiple first segments.

[0176] The memory 820 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 8201 and / or cache memory 8202, and may further include read-only memory (ROM) 8203.

[0177] The memory 820 may also include a program / utility 8204 having a set (at least one) of program modules 8205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0178] Bus 830 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0179] Electronic device 800 can also communicate with one or more external devices 840 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more first text processors that enable user interaction with electronic device 800, and / or with any device that enables electronic device 800 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 850. Furthermore, electronic device 800 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 860. As shown, network adapter 860 communicates with other modules of electronic device 800 via bus 830. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 800, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0180] In the disclosed exemplary embodiments, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium.

[0181] In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the foregoing “Detailed Description” section of this specification according to various exemplary embodiments of this disclosure.

[0182] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0183] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0184] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0185] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on a terminal device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0186] This disclosure provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform a text processing method provided in various alternative embodiments of this disclosure.

[0187] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0188] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0189] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0190] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope of this disclosure is indicated by the appended claims.

Claims

1. A text processing method characterized by, The method comprises the following steps: receiving text information and user question information about the text information; dividing the text information into multiple sentences, generating a feature vector of each sentence, and calculating the similarity between the feature vectors of any two adjacent sentences in the text information; determining a breakpoint set of the text information based on the similarity between the feature vectors of any two adjacent sentences; performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first segments; generating reply information of the question information based on the multiple first segments.

2. The method of claim 1, wherein, The method comprises the following steps: based on the similarity between the feature vectors of any two adjacent sentences, calculating the similarity rate in each window through a sliding window on the text information; based on the similarity rate in each window, determining the breakpoint set of the text information.

3. The method of claim 1, wherein, Before the method of performing density clustering on the multiple sentences based on the breakpoint set to obtain multiple first segments, the method further comprises the following steps: constructing a semantic graph corresponding to the text information by taking each sentence as a graph node and taking the similarity between any two adjacent sentences as the edge weight of the corresponding graph nodes of the two adjacent sentences; optimizing the breakpoint set based on the semantic graph using a graph neural network.

4. The method of claim 1, wherein, The method comprises the following steps: based on the fragmentation granularity, merging and splitting the multiple first segments to obtain multiple second segments, wherein the fragmentation granularity is the minimum and maximum Token number of a segment; determining a target segment corresponding to the question information from the multiple second segments; generating reply information of the question information based on the target segment.

5. The method of claim 4, wherein, Before the method of merging and splitting the multiple first segments based on the fragmentation granularity to obtain multiple second segments, the method further comprises the following steps: determining the content complexity of the text information, wherein the content complexity includes syntax complexity, semantic density, and logical complexity; obtaining real-time resource information of hardware; based on the content complexity and the real-time resource information, determining the fragmentation granularity.

6. The method of claim 5, wherein, The method comprises the following steps: based on the content complexity, determining the fragmentation granularity using a multilayer perceptron; based on the real-time resource information, dynamically adjusting the fragmentation granularity.

7. The method of claim 5, wherein, The method comprises the following steps: generating a syntax dependency tree for each sentence; based on the syntax dependency tree of each sentence, determining the syntax complexity; based on the entity frequency and term frequency of the text information, determining the semantic density; based on the argument hierarchy number of the text information, determining the logical complexity.

8. The method of claim 1, wherein, The method further comprises the following steps: taking the feature vector of each sentence, the similarity between the feature vectors of any two adjacent sentences, and the real-time resource information of hardware as the state, determining the action through a policy network, wherein the action includes the value of the fragmentation granularity and whether to split; based on the action, splitting the text information into multiple third segments; generate the reply information based on the plurality of third fragments.

9. The method of claim 8, wherein, After the reply information is generated based on the action, the method further includes: determining a question-and-answer accuracy rate of the reply information, obtaining a satisfaction degree of the user regarding the reply information, and resource consumption for generating the reply information; determining a reward value based on the question-and-answer accuracy rate, the satisfaction degree, and the resource consumption; calculating a loss value using a loss function based on the reward value; optimizing the policy network based on the loss value.

10. The method of claim 9, wherein, After the policy network is optimized based on the loss value, the method further includes: optimizing a multi-layer perception and a window size of a sliding window using the policy network, wherein the multi-layer perception is used to determine the fragmentation granularity, the fragmentation granularity is related to generating the reply information, and the sliding window is used to determine a similarity change rate, the similarity change rate is related to determining the breakpoint set.

11. A text processing apparatus characterized by comprising: comprise: a receiving unit configured to receive text information and question information of the user regarding the text information; a processing unit configured to divide the text information into a plurality of sentences, generate a feature vector of each sentence, and calculate a similarity between feature vectors of any two adjacent sentences in the text information; a determining unit configured to determine a breakpoint set of the text information based on the similarity between the feature vectors of any two adjacent sentences; a clustering unit configured to perform density clustering on the plurality of sentences based on the breakpoint set to obtain a plurality of first fragments; a generating unit configured to generate reply information of the question information based on the plurality of first fragments.

12. An electronic device, comprising: comprise: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the method of any one of claims 1-10 by executing the executable instructions.

13. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method of any one of claims 1-10.

14. A computer program product comprising computer instructions stored in a computer readable storage medium, the computer instructions being executed by a processor to implement the operational instructions of the method of any one of claims 1-10.

Images