Text processing method and device, computer equipment and storage medium

Abstract

The invention relates to the field of text processing, and discloses a text processing method and device, computer equipment and a storage medium, and the method comprises the steps that at least one target text vector is acquired from a vector database according to a query request, the vector database comprises a plurality of text vectors, and the text vectors are obtained according to text slicing; obtaining a target text slice corresponding to the target text vector, and a front-order text slice and a rear-order text slice adjacent to the target text slice; and according to the target text slice, the front-order text slice and the back-order text slice, constructing and restoring the document content. According to the method and the device, the problem of incomplete semantics caused by text slicing in the prior art is solved.

Description

Technical Field

[0001] This invention relates to the field of text processing, and more specifically to a text processing method, apparatus, computer device, and storage medium. Background Technology

[0002] In practical applications of large-scale models, Retrieval-Augmented Generation (RAG) is an important means for large models to acquire dynamic information. Due to the limitations of the context window of large models, it is usually necessary to split long corpora into smaller semantic segments according to a certain length, and then select some segments for processing by the large model based on the semantic relevance of the user query. However, this text slicing method can lead to serious semantic fragmentation problems, destroying the coherence of the original text and affecting the large model's understanding and generation quality of the overall semantics.

[0003] Existing technologies primarily employ two approaches to address semantic segmentation: one is to preserve overlapping content between paragraphs during text slicing, and the other is to perform semantic segmentation based on the table of contents and chapters. However, preserving overlapping content increases the length of segments and semantic complexity, affecting not only the accuracy of vector retrieval but also increasing the difficulty of understanding large models and the consumption of segments. On the other hand, the segmentation method based on the table of contents and chapters has significant limitations in applications such as documents lacking a clear table of contents structure or chapter lengths exceeding the processing capacity of vector models, and still cannot avoid the semantic segmentation problem. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a text processing method, apparatus, computer device, and storage medium to solve the problem of semantic incompleteness caused by text slicing in the prior art.

[0005] In a first aspect, embodiments of the present invention provide a text processing method, the method comprising:

[0006] At least one target text vector is retrieved from a vector database according to a query request, wherein the vector database includes multiple text vectors, and the text vectors are obtained based on text slicing; Obtain the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice; The document content is reconstructed based on the target text slice, the preceding text slice, and the following text slice.

[0007] Furthermore, before retrieving multiple text vectors from the vector database according to the query request, the method further includes: Retrieve the original document content and its corresponding document identifier; The original document content is segmented according to preset segmentation rules to obtain multiple text slices; Based on the segmentation order corresponding to multiple text slices, generate an order identifier associated with each text slice, and construct a corresponding text vector based on the semantic features of the text slices; Each text slice is associated with the sequence identifier and the document identifier and stored in the slice relationship table, and the text vector corresponding to each text slice is associated with the sequence identifier and the document identifier and stored in the vector database.

[0008] Furthermore, obtaining at least one target text vector from the vector database according to the query request includes: Parse the retrieval semantic vector in the query request; Calculate the similarity between each text vector in the vector database and the retrieved semantic vector; At least one target text vector is obtained from the vector database based on the similarity.

[0009] Furthermore, obtaining the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice, includes: Obtain the target document identifier and target sequence identifier associated with the target text vector; The corresponding target text slice is retrieved from the slice relationship table using the target document identifier and the target sequence identifier; Query the preceding and following slice sequences corresponding to the target text slice in the slice relationship table; A first number of preceding text slices are extracted from the preceding slice sequence in reverse order, and a second number of following text slices are extracted from the following slice sequence in forward order.

[0010] Furthermore, the first quantity and / or the second quantity are determined based on at least one of the following methods: Receives a first quantity specified by the user for the preceding slice and a second quantity specified for the subsequent slice; The required number of segments is determined based on the semantic completeness of the target text slice, and a first number for the preceding slice and a second number for the subsequent slice are calculated based on the required number of segments. The first number of slices and the second number of slices are calculated based on the document length of the original document content and preset ratio parameters.

[0011] Furthermore, the step of constructing and restoring the document content based on the target text slice, the preceding text slice, and the following text slice includes: The preceding text slice, the target text slice, and the following text slice are deduplicated to obtain a deduplicated text slice. The deduplicated text slices are sorted according to the sequence identifier to generate continuous slice content; The document structure is reconstructed based on the original segmentation positions of the continuous slice content, and the restored document content is generated.

[0012] Furthermore, the step of reconstructing the document structure based on the original segmentation positions of the continuous slice content to generate the restored document content includes: Identify the semantic coherence of the original segmentation positions within the continuous slice content; Based on the semantic coherence, structured elements are added to the continuous slice content to generate the restored document content.

[0013] Secondly, embodiments of the present invention provide a text processing apparatus, the apparatus comprising: The first acquisition module is used to acquire at least one target text vector from a vector database according to a query request, wherein the vector database includes multiple text vectors, and the text vectors are obtained based on text slicing; The second acquisition module is used to acquire the target text slice corresponding to the target text vector, as well as the preceding text slice and the following text slice adjacent to the target text slice; A construction module is used to construct and restore document content based on the target text slice, the preceding text slice, and the following text slice.

[0014] Thirdly, embodiments of the present invention provide a computer device, including: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the method described in the first aspect or any corresponding embodiment thereof.

[0015] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the method described in the first aspect or any corresponding embodiment thereof.

[0016] The method provided in this application has the following beneficial effects: The method provided in this application retrieves the target text vector from a vector database based on the query request, and uses vectorized retrieval to accurately locate text fragments semantically related to the user's query, ensuring the core relevance of the retrieval results and avoiding the inefficiency caused by traditional full-text scanning or keyword matching. It acquires the target text slice and its adjacent preceding and succeeding text slices, and dynamically expands the retrieval context by associating the slice order information stored in a relational database. Without relying on pre-set overlapping content or a fixed directory structure, it effectively restores the semantic coherence broken by forced slicing, solving the problem of information fragmentation. Based on the target slice and adjacent slices, it constructs and reconstructs the document content, reorganizing the discrete retrieval results into logically coherent semantic units, providing complete contextual input materials for downstream large models, improving the quality and accuracy of the generated content, and reducing the additional computational overhead caused by content duplication or structural chaos. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating a text processing method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating another text processing method according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating another text processing method according to an embodiment of the present invention; Figure 4 This is a structural block diagram of a text processing apparatus according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] According to embodiments of the present invention, a text processing method, apparatus, computer device, and storage medium are provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0021] This embodiment provides a text processing method. Figure 1 This is a flowchart of a text processing method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain at least one target text vector from the vector database according to the query request. The vector database includes multiple text vectors, which are obtained from text slicing.

[0022] In this embodiment, a user-input query request is received, converted into a retrieval semantic vector using a semantic encoding model, and then a similarity retrieval is performed in a vector database. Specifically, an approximate nearest neighbor search algorithm can be used to compare the similarity between the retrieval semantic vector and all text vectors in the database. The most relevant set of target text vectors is selected based on a preset similarity threshold and ranking mechanism. When the query request contains keyword combinations, semantic vectors for each keyword are generated separately and then weighted and fused for retrieval. At the same time, the similarity threshold is automatically adjusted according to the query complexity (e.g., a higher threshold is used for simple queries to ensure accuracy, while the threshold is appropriately lowered for complex queries to expand the recall range).

[0023] Step S102: Obtain the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice.

[0024] In this embodiment, based on the target text vector, the target document identifier and target sequence identifier are obtained by parsing its associated metadata. Then, the target text slice is queried in the slice relationship table. In the slice relationship table, using the target document identifier as the filtering condition, all records with a value less than the target sequence identifier are queried to form the preceding slice sequence, and all records with a value greater than the target sequence identifier are queried to form the following slice sequence. When the target slice is located at the beginning of the document, the acquisition of the preceding slice is automatically skipped, and when it is located at the end of the document, the acquisition of the following slice is skipped. The acquisition range of adjacent slices is automatically expanded or reduced according to the semantic density of the slice content. By extracting a specified number of adjacent slices from the sequence, a complete fragment set containing the target slice and its context is formed.

[0025] Step S103: Construct the restored document content based on the target text slice, the preceding text slice, and the following text slice.

[0026] In this embodiment, all obtained text slices are first deduplicated by hash comparison and semantic similarity analysis to eliminate duplicate content. Then, the deduplicated slices are sorted and recombined according to their sequence identifiers to generate continuous slice content that maintains the original document order. By analyzing the semantic coherence between slices, paragraph boundaries are automatically identified, and the document hierarchy is restored based on the segmentation position of the original document, including adding structured elements such as paragraph identifiers and chapter titles. When slices from multiple source documents are involved, they can be grouped and reconstructed according to document source and semantic relevance to ensure that the final generated restored document content maintains both semantic integrity and good readability.

[0027] The method provided in this application retrieves the target text vector from a vector database based on the query request, and uses vectorized retrieval to accurately locate text fragments semantically related to the user's query, ensuring the core relevance of the retrieval results and avoiding the inefficiency problems caused by traditional full-text scanning or keyword matching. It obtains the target text slice and its adjacent preceding and succeeding text slices, and dynamically expands the retrieval context by associating the slice order information stored in a relational database. Without relying on preset overlapping content or fixed directory structure, it effectively restores the semantic coherence broken by forced slicing and solves the problem of information fragmentation. Based on the target slice and adjacent slices, it constructs a logically coherent restored document content.

[0028] The restored document content can be used as input data into the large model, enabling the large model to generate coherent response content based on the complete restored document content, thus avoiding erroneous reasoning or content fragmentation caused by missing information or semantic breaks.

[0029] In this embodiment of the application, obtaining at least one target text vector from a vector database according to a query request includes: Step A1: Parse the retrieval semantic vector in the query request.

[0030] Specifically, firstly, the system receives a query request (e.g., "development of new energy battery technology") input by the user in natural language. Then, it performs real-time vectorization processing on the query request using a pre-trained semantic encoding model (such as BERT, SBERT, or other text embedding models). This process includes: segmenting the query text and performing semantic understanding to generate a high-dimensional dense vector (i.e., a retrieval semantic vector). This vector numerically represents the deep semantic features of the query statement. During this process, a vectorization interface deployed locally or in the cloud is invoked to map the variable-length natural language query into a fixed-dimensional semantic vector representation (e.g., a 768-dimensional or 1024-dimensional floating-point array). This retrieval semantic vector serves as the benchmark anchor for subsequent similarity calculations.

[0031] Step A2: Calculate the similarity between each text vector in the vector database and the retrieved semantic vector.

[0032] Specifically, the process first loads the retrieval semantic vector (i.e., the numerical representation of the query request) and then iterates through each text vector stored in the vector database (i.e., a high-dimensional dense vector obtained by vectorizing text slices in advance through a semantic encoding model, with each vector corresponding to the semantic features of a text segment). Then, the cosine similarity algorithm is used as the core calculation method to quantify the semantic similarity between the retrieval semantic vector and each text vector by calculating the cosine value of the angle between them. This value ranges from [-1, 1], with the closer to 1 indicating greater semantic similarity. In the actual calculation process, multiple vector pairs are processed in parallel, and the Approximate Nearest Neighbor (ANN) search technique is used to optimize the scanning performance of the large-scale vector database, ensuring that the similarity calculation is completed within an acceptable time.

[0033] Step A3: Obtain at least one target text vector from the vector database based on similarity.

[0034] Specifically, the process begins with a preliminary screening based on a preset similarity threshold (e.g., 0.75) to filter out irrelevant vectors with excessively low similarity. Next, a Top-K sorting mechanism is used to rank the remaining text vectors by similarity score from highest to lowest, selecting the top K (e.g., K=5) most similar text vectors as the candidate set. In practical applications, the K value can be dynamically adjusted, either by setting a fixed parameter or by adaptively determining the value based on the distribution of similarity scores (e.g., only the top 3 are selected when the highest similarity exceeds 0.9, otherwise the top 10 are selected). Finally, the selected text vectors and their associated document identifiers and sequence identifiers are encapsulated into a target text vector set for output, providing a semantic matching basis for content segmentation reconstruction.

[0035] The system parses the retrieval semantic vector in the query request, transforming the user's intent into a computable semantic representation; it calculates vector similarity and ensures the objectivity of semantic matching through quantization comparison; it obtains the target text vector based on the similarity and uses threshold filtering and sorting mechanisms to ensure that the retrieval results have both semantic relevance and priority order, providing a high-quality initial matching set for subsequent context reconstruction.

[0036] In this embodiment of the application, obtaining the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice, includes: Step B1: Obtain the target document identifier and target sequence identifier associated with the target text vector.

[0037] Specifically, by parsing the metadata information associated with each target text vector when it is stored in the vector database, the target document identifier and target sequence identifier contained therein are extracted. The target document identifier is a string that uniquely identifies the original document (e.g., "EV2025001"), while the target sequence identifier is the numerical sequence number assigned to the text slice during the original document segmentation process (e.g., 1, 2, 3). In the implementation, when the vector database stores text vectors, it simultaneously saves the vector data and its associated metadata in key-value pairs, where the key is a combination of "doc_id:sequence_id" and the value is the text vector data. By calling the metadata query interface provided by the vector database, the corresponding key value is directly obtained based on the internal pointer of the target text vector, and then the target document identifier and target sequence identifier are separated through string parsing. For batch processing scenarios, a parallel query mechanism is used to process the metadata extraction of multiple target text vectors simultaneously, and a temporary hash table is established to maintain the mapping relationship of "text vector-document identifier-sequence identifier" to ensure that subsequent steps can quickly access this associated information.

[0038] Step B2: Use the target document identifier and the target order identifier to query the corresponding target text slice from the slice relationship table.

[0039] Specifically, a query request is initiated to the slice relation table based on the target document identifier and the target sequence identifier. The slice relation table is a structured data table established in a relational database that stores the mapping relationship between "document identifier - sequence identifier - text slice," and its primary key is usually composed of both the document identifier and the sequence identifier. During execution, a structured query statement (such as an SQL statement) is constructed, substituting the target document identifier and the target sequence identifier as query parameters, and retrieving the text slice content from the corresponding records in the slice relation table. A batch query optimization strategy can also be adopted. When multiple combinations of target document identifiers and sequence identifiers exist, multiple slice contents can be queried at once using the "IN" condition. Simultaneously, a query result caching mechanism is established to cache frequently accessed slice content, avoiding repeated queries on the same slice.

[0040] Step B3: Query the preceding and following slice sequences corresponding to the target text slice in the slice relationship table.

[0041] Specifically, based on the acquired target document identifier and target sequence identifier, a range query is performed in the slice relationship table to obtain the complete adjacent slice sequence. The preceding slice sequence refers to the set of all text slices with a sequence identifier less than the target sequence identifier under the same document identifier, arranged in ascending order by number. The following slice sequence refers to the set of all text slices with a sequence identifier greater than the target sequence identifier under the same document identifier, arranged in ascending order by number. Two independent directional query statements are constructed: for the preceding slice sequence, it is sorted in descending order by sequence identifier to ensure priority for the nearest preceding slice; for the following slice sequence, it is sorted in ascending order by sequence identifier to maintain the natural order. Additionally, a join query optimization strategy can be used to simultaneously obtain sequence data from both directions using a single SQL statement, and database indexes (such as composite indexes based on document identifier and sequence identifier) ​​can be used to improve query efficiency. For massive document scenarios, a sharding query mechanism can also be supported, decomposing the sequence query of large documents into multiple subqueries for parallel processing.

[0042] Step B4: Extract the first number of preceding text slices from the preceding slice sequence in reverse order, and extract the second number of following text slices from the following slice sequence in forward order.

[0043] Specifically, based on the preceding and following slice sequences, slice extraction is performed according to preset quantity control parameters. The first quantity refers to the number of preceding slices to be obtained, and the second quantity refers to the number of following slices to be obtained. These two quantity parameters can be predetermined through various methods such as user specification, semantic integrity analysis, or document length ratio calculation. For the preceding slice sequence (which is arranged in descending order by its sequence identifier), the first N elements (N being the first quantity) are extracted using a list truncation operation to obtain the N preceding text slices closest to the target slice. For the following slice sequence (which is arranged in ascending order by its sequence identifier), the first M elements (M being the second quantity) are also extracted using a list truncation operation to obtain the M following text slices. Furthermore, when the requested number of slices exceeds the actual length of the sequence, the extraction quantity is automatically adjusted to the maximum available value to ensure that no null values ​​or errors are returned. A weighted extraction strategy is also supported, evaluating the semantic density of the slices in the sequence and prioritizing the extraction of slices with greater information content.

[0044] By acquiring the target document identifier and target sequence identifier associated with the target text vector, the binding of semantic vectors and document structure metadata was achieved, establishing a mapping channel from high-dimensional vector space to structured text space, providing a crucial foundation for the precise location and subsequent processing of target content. By utilizing the target document identifier and target sequence identifier to query the corresponding target text slice from the slice relationship table, content retrieval based on dual identifiers was achieved, ensuring the complete acquisition of core semantic content. By querying the preceding and following slice sequences corresponding to the target text slice in the slice relationship table, the complete contextual environment of the original document was systematically reconstructed, restoring the semantic association network broken by text segmentation. Finally, by extracting a specified number of preceding text slices from the preceding slice sequence in reverse and a specified number of following text slices from the following slice sequence in forward, intelligent control and dynamic adjustment of the context scope were achieved, fully restoring semantic coherence while effectively avoiding information redundancy, ultimately forming a text unit that is structurally complete, logically rigorous, and semantically coherent.

[0045] In the embodiments of this application, the first quantity and / or the second quantity are determined based on at least one of the following methods: receiving a first quantity specified by the user for the preceding slice and a second quantity specified for the subsequent slice; determining the segment quantity requirement based on the semantic integrity of the target text slice, and calculating the first quantity for the preceding slice and the second quantity for the subsequent slice based on the segment quantity requirement; calculating the first quantity for the preceding slice and the second quantity for the subsequent slice based on the document length of the original document content and a preset ratio parameter.

[0046] Specifically, there are three methods: Method 1: The system receives the first and second quantities (number of preceding and following slices) directly specified by the user through the interactive interface. This method uses a parameter validation module to ensure that the input values ​​are within a preset valid range (e.g., 1-10) and supports setting differentiated quantity permissions for different user roles. Method 2: The system automatically determines the required number of slices based on the semantic completeness of the target text slices. By analyzing the sentence structure completeness (e.g., distribution of segmentation symbols), semantic boundary features (e.g., density of topic transition words), and contextual dependencies of the target text slices, a pre-trained completeness assessment model is used to calculate the semantic completeness score (range 0-1). When the score is below a threshold (e.g., 0.6), the number of adjacent slices is automatically expanded according to a default ratio (e.g., one adjacent slice is added for every 0.1 decrease in completeness). Method 3: The system dynamically calculates the quantity based on the total number of slices in the original document content and a preset ratio parameter (e.g., 5%). The total number of slices in the document is obtained by querying the slice relationship table, and the baseline quantity is obtained by rounding "total number of slices × ratio parameter". This is then adaptively adjusted based on the current target slice's position information (e.g., disabling preceding slice extraction when the slice is at the beginning of the document).

[0047] During the integration process, a three-tier priority strategy is established: Documents explicitly specified by the user are given the highest priority, and their input values ​​are directly used; when the user does not specify a priority, semantic completeness assessment and document proportion calculation are performed sequentially, and a final quantity value is generated through a weighted decision model (e.g., completeness weight 0.7, proportion parameter weight 0.3). Historical adjustment record analysis is also supported. By statistically analyzing users' actual usage preferences for different document types, an adaptive quantity recommendation model is established to continuously optimize default parameter settings.

[0048] The system receives a specified number of data points from the user, providing personalized interaction capabilities; determines the number based on semantic integrity, enabling adaptive context expansion; and calculates the number based on document length ratios, ensuring the rationality of context acquisition. The collaborative work of these three methods allows the system to dynamically optimize the context range according to different scenarios, balancing semantic integrity and processing efficiency.

[0049] In this embodiment of the application, constructing and restoring document content based on the target text slice, the preceding text slice, and the following text slice includes: Step C1 involves deduplicating the preceding text slice, the target text slice, and the subsequent text slice to obtain the deduplicated text slice.

[0050] Specifically, the system receives an original set of text slices consisting of preceding text slices, target text slices, and subsequent text slices. It then eliminates duplicate content through multiple deduplication mechanisms: First, a fast comparison mechanism based on slice content hash values ​​is established, calculating the MD5 or SHA-1 hash value for each text slice and detecting and removing identical duplicate slices using a hash table. Next, semantic-level deduplication is performed, using a semantic similarity calculation model (such as Sentence-BERT) to identify and filter approximately duplicate slices with a semantic repetition rate exceeding a preset threshold (e.g., 95%). Finally, a context-aware deduplication strategy is included, analyzing the slices within the original document. Positional relationships: When adjacent slices partially overlap due to expansion operations, intelligent cropping is performed based on slice boundary information to retain the most complete semantic units; in addition, configurable deduplication granularity is supported, allowing users to choose between strict deduplication (retaining only completely unique slices) or lenient deduplication (allowing a certain degree of content overlap) based on their settings; all deduplication operations retain the first occurrence of the slice instance to ensure that semantic coherence is not compromised by deduplication; the final output deduplicated text slice set eliminates unnecessary duplicate content while maximizing the integrity of the original semantic information, providing a clean data foundation for subsequent sorting and document reconstruction.

[0051] Step C2: Sort the deduplicated text slices according to the sequence identifier to generate continuous slice content.

[0052] Specifically, for a deduplicated text slice set, a multi-level sorting system is constructed by parsing the sequence identifier and document identifier associated with each slice, using the document identifier as the primary sorting key and the sequence identifier as the secondary sorting key. A stable sorting algorithm (such as merge sort) is used to rearrange all text slices, ensuring that slices under the same document identifier are strictly arranged in ascending order according to the numerical value of the sequence identifier, forming continuous slice content that conforms to the original document order. When a break or repetition in the sequence identifier is detected, an auxiliary sorting mechanism based on the semantic coherence of the slice content is automatically activated. This mechanism optimizes the slice arrangement order by analyzing the semantic coherence of adjacent slices (such as using referential resolution and topic coherence analysis). It also supports the orderly integration of cross-document slices. When slices involve multiple documents, overall sorting optimization can be performed based on document weight or slice relevance. For large-scale slice sets, a segmented sorting and merging strategy is adopted: slices are first grouped and sorted by document identifier, and then cross-group merging is performed. The final continuous slice content maintains the semantic flow and logical structure of the original document, providing a correct and orderly text foundation for subsequent document structure reconstruction.

[0053] Step C3: Reconstruct the document structure based on the original segmentation positions of the continuous slice content to generate the restored document content.

[0054] Deduplication eliminates content redundancy and reduces redundant computations in large models; sorting by sequence identifier restores the original logical flow of the text; reconstructing the document structure improves the readability and structure of the content. The final generated restored document content maintains semantic coherence and has a good organizational form, providing high-quality input for large model processing.

[0055] In this embodiment of the application, reconstructing the document structure based on the original segmentation positions of continuous slice content to generate restored document content includes: identifying the semantic coherence of the original segmentation positions in the continuous slice content; adding structured elements to the continuous slice content based on semantic coherence to generate restored document content.

[0056] Specifically, the system first performs semantic coherence analysis on consecutive slices. A pre-trained language model identifies semantic association features between adjacent slices at their original segmentation positions, including but not limited to consistency of referential relationships, frequency of logical connectors, strength of topic continuity, and paragraph boundary probability. At the micro level, it detects the connection between sentences (e.g., using core entity tracking technology), and at the macro level, it analyzes the topic evolution path between paragraphs (e.g., through TF-IDF weighted keyword distribution changes). Based on the identified semantic coherence results, structured elements are dynamically added: section markers are inserted when strong semantic transitions are detected, bullet points are added when enumeration relationships are identified, and paragraph indentation is reconstructed when argumentative structures are found. Furthermore, it can automatically match suitable set of structured elements by selecting the corresponding document template based on the original document type (e.g., technical report, academic paper, or news article); simultaneously, it recovers a multi-level heading system by analyzing heading-level features and intelligently divides paragraph lengths based on semantic density distribution. The final generated restored document content not only maintains the semantic integrity of the original document but also enhances the readability and logic of the content through the reconstructed document structure, providing high-quality structured input for large-scale model processing.

[0057] By identifying semantic coherence, the inherent logical relationships of the text are accurately captured; by adding structured elements, the hierarchical structure of the document is restored based on the semantic analysis results, so that the final output document is not only coherent in content, but also has a clear structure, which is more in line with reading habits and the needs of large model processing.

[0058] In the embodiments of this application, such as Figure 2 As shown, before retrieving multiple text vectors from the vector database based on the query request, the method further includes: Step S201: Obtain the original document content and the corresponding document identifier.

[0059] In this embodiment, the original document content is acquired from local storage systems, cloud document libraries, or network resources through a multi-source acquisition module. This original document content refers to the complete text data to be processed, supporting multiple formats including PDF, Word, TXT, and HTML. Simultaneously, a corresponding document identifier is created using a document identifier generator. This identifier is a globally unique string composed of "organization code + timestamp + random sequence" (e.g., "ABC_20250010020030_5A6B7C"), ensuring uniqueness even in a distributed environment. First, the original document undergoes format parsing and text extraction, retaining the text content and basic metadata (such as creation time, author, and title). Then, a digital fingerprint is calculated for the document content using a hash algorithm (e.g., SHA-256), and this fingerprint is bound to the generated document identifier and stored in the document registry. Furthermore, when a new document's digital fingerprint is detected to be duplicated with an existing document, the existing document identifier is automatically reused. Batch document acquisition mode is also supported, automatically triggering the document acquisition process by monitoring file changes in a specified directory, and optimizing parallel processing for large document sets.

[0060] Step S202: The original document content is segmented according to preset segmentation rules to obtain multiple text slices.

[0061] In this embodiment, the original document content is intelligently segmented based on preset segmentation rules. These preset segmentation rules include, but are not limited to, fixed-length segmentation (e.g., each slice consists of 512 characters), semantic unit segmentation (e.g., segmentation by paragraph or chapter title), and segmentation based on specific delimiters (e.g., segmentation by period or page break). The implementation first preprocesses the original document content, including standardizing the encoding format and removing irrelevant formatting symbols. Then, the segmentation operation is performed according to the selected segmentation rules: when fixed-length segmentation is used, a sliding window mechanism ensures that the number of characters in each slice precisely matches the set value, and fine-tuning is performed when a complete sentence is interrupted to avoid semantic fragmentation; when semantic unit segmentation is used, natural language processing technology is used to identify paragraph boundaries and chapter titles in the document, and a pre-trained layout analysis model is used to accurately detect document structural elements. In addition, the system automatically selects the optimal segmentation rules by analyzing document type (e.g., technical documents are segmented by chapter, and press releases are segmented by paragraph), and supports mixed segmentation modes (e.g., segmenting by chapter first and then applying fixed-length subdivisions to excessively long chapters). At the same time, it preserves the original position information of each text slice and its relationship with adjacent slices, laying the foundation for subsequent sequence identifier allocation and semantic coherence restoration. The final output of multiple text slices maintains the semantic independence of the original document while ensuring retrieval efficiency and content restoration quality in subsequent processing stages.

[0062] Step S203: Generate a sequence identifier associated with each text slice based on the segmentation order corresponding to multiple text slices, and construct the corresponding text vector according to the semantic features of the text slices.

[0063] In this embodiment, each text slice is first assigned a sequentially increasing sequence identifier (e.g., an integer sequence starting from 1) according to its natural order of appearance in the original document. This sequence identifier, together with the document identifier, constitutes the unique positional fingerprint of the slice. When constructing the text vector, a pre-trained semantic encoding model (e.g., Sentence-BERT, SimCSE, etc.) is used to perform deep semantic representation on each text slice. The specific process includes: after segmenting the slice text, it is input into an encoder network, and context-aware semantic features are extracted through a multi-layer Transformer structure. Finally, a fixed-dimensional text vector (e.g., a 768-dimensional floating-point array) is output through a pooling layer. When processing domain-specific documents, the system automatically loads domain-adapted semantic models (e.g., BioBERT for medical literature and Legal-BERT for legal documents) and ensures scale consistency across different text slices through vector normalization. Simultaneously, it records the generation parameters of each text vector (including model version, vector dimension, and timestamp) to establish a complete vector source archive. For long text slices, it supports segmented encoding and feature fusion strategies, first encoding the segments within each slice and then weighting and aggregating them through an attention mechanism, effectively improving the accuracy of semantic representation of long texts. The resulting sequence identifiers and text vectors provide crucial structured data and semantic indexing foundations for subsequent vector retrieval and content reconstruction.

[0064] Step S204: Associate each text slice with its sequence identifier and document identifier and store it in the slice relationship table; associate the text vector corresponding to each text slice with its sequence identifier and document identifier and store it in the vector database.

[0065] In this embodiment, a slice relationship table is first constructed. This table uses a three-column structure in a relational database (such as MySQL or PostgreSQL): document identifier (as a foreign key associated with the original document), sequence identifier (an integer type recording the slice position), and text slice (a text type storing the complete fragment). A composite primary key (document identifier - sequence identifier) ​​is used to ensure data uniqueness, and a B-tree index is built to optimize query performance. At the same time, text vectors are associated with their corresponding document identifiers and sequence identifiers as metadata and stored in a vector database (such as FAISS or Chroma). The data is organized in key-value pairs, where the key is a combination string of "document identifier: sequence identifier" and the value is a floating-point array of text vectors. The indexing mechanism of the vector database (such as HNSW graph index) is used to achieve similarity search. In addition, a two-phase commit protocol ensures the atomicity of writing to the slice relationship table and vector database; it also supports batch storage optimization, packaging multiple text slices and their vectors into batches and writing them in parallel, improving the processing efficiency of large-scale documents; furthermore, it can also integrate a storage monitoring module to track storage status in real time and automatically handle anomalies (such as duplicate identifier conflicts or vector dimension mismatches), ensuring that all data is intact and persistent, providing a reliable data foundation for subsequent retrieval stages.

[0066] It should be noted that, as Figure 3 As shown, the text processing method includes a slicing and retrieval restoration process: First, the original document is sliced ​​into multiple text slices according to technical classification; then, the document ID and slice order are recorded, and a semantic vector for each slice is generated through vector model processing; the document ID and corresponding vector value are stored in a vector database, and the mapping relationship between slice content, sequence identifier, and document ID is stored in a slice relationship table; when a user initiates a query request, the query request is transformed into a retrieval semantic vector, and a similarity retrieval is performed in the vector database; the sequence number and document ID of semantically similar target slices are obtained; after rearrangement filtering, the slice relationship table is searched according to the sequence number and document ID; the content of adjacent slices is obtained according to a specified number; the obtained slices are deduplicated according to the sequence number and rearranged in natural order; finally, the restored coherent content is returned.

[0067] Obtaining the original document content and corresponding document identifiers provides a traceable data source for subsequent processing; segmentation according to preset segmentation rules ensures the semantic rationality of text slices; generating sequence identifiers and constructing text vectors establishes a dual index of position and semantics for the slices; associative storage in the slice relationship table and vector database forms a well-defined storage system, enabling the vector database to specialize in semantic retrieval and the slice relationship table to be responsible for content restoration. The collaboration between the two significantly improves the efficiency and accuracy of subsequent retrieval stages.

[0068] As an example, suppose we have two parsed long documents: "2025 New Energy Vehicle Technology Report" (document ID: EV2025001) and "Overview of New Energy Battery R&D Progress" (document ID: BAT2025002). First, the two documents are segmented: EV2025001 is segmented into three sections based on "technology classification": "Improvement of Power Battery Energy Density," "Integration and Application of Autonomous Driving," and "Breakthroughs in Vehicle-to-Everything (V2X) Technology," numbered 1, 2, and 3 respectively. BAT2025002 is segmented into three sections with a fixed length: "Optimization of Ternary Lithium Battery Cycle Life," "Difficulties in Mass Production of Solid-State Batteries," and "Innovation in Sodium-Ion Battery Materials," numbered 1, 2, and 3 respectively. Then, the correspondence between "document ID - section sequence number - section content" is stored in a section relationship table. Simultaneously, each section is vector-encoded, and the document ID and its corresponding text vector are stored in a vector database.

[0069] When a user queries "development of new energy battery technology", the semantic vector database is retrieved based on the query's semantic vector. Two semantically similar slices are matched: slice 1 of EV2025001 (original text "improvement of power battery energy density") and slice 1 of BAT2025002 (original text "optimization of ternary lithium battery cycle life"), and their respective document IDs and slice sequence numbers are obtained. Next, based on these two sets of document IDs and slice sequence numbers, the slice relationship table is searched. According to the user's request to "return one related slice before and after", slice 1 (no preceding slice) and slice 2 (subsequent "autonomous driving integration application") of EV2025001, and slice 1 (no preceding slice) and slice 2 (subsequent "difficulties in solid-state battery mass production") of BAT2025002 are returned. Finally, these contents are deduplicated and sorted by "document ID + slice sequence number" to restore the original content, ultimately returning the coherent content of "improvement of power battery energy density → autonomous driving integration application" and "optimization of ternary lithium battery cycle life → difficulties in solid-state battery mass production".

[0070] This example clearly demonstrates the core architecture of this invention: the vector database stores document IDs and slice text vectors, supporting semantic retrieval and outputting semantically similar slice original texts and their metadata; the slice relationship table stores complete slice content mapping relationships, providing specific content query basis for vector retrieval results. The two are linked through "document ID - slice sequence number," collaboratively achieving a complete technical solution from semantic query to accurate content restoration.

[0071] This embodiment also provides a text processing apparatus for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0072] This embodiment provides a text processing device, such as... Figure 4 As shown, it includes: The first acquisition module 41 is used to acquire at least one target text vector from the vector database according to the query request, wherein the vector database includes multiple text vectors, and the text vectors are obtained based on text slicing; The second acquisition module 42 is used to acquire the target text slice corresponding to the target text vector, as well as the preceding text slice and the following text slice adjacent to the target text slice; Module 43 is used to construct and restore the document content based on the target text slice, the preceding text slice, and the following text slice.

[0073] In this embodiment of the application, the device further includes: a storage module, used to acquire the original document content and the corresponding document identifier; segment the original document content according to a preset segmentation rule to obtain multiple text slices; generate a sequence identifier associated with each text slice based on the segmentation order corresponding to the multiple text slices, and construct a corresponding text vector according to the semantic features of the text slices; associate and store each text slice with the sequence identifier and the document identifier in a slice relationship table, and associate and store the text vector corresponding to each text slice with the sequence identifier and the document identifier in a vector database.

[0074] In this embodiment of the application, the first acquisition module 41 is specifically used to parse the retrieval semantic vector in the query request; calculate the similarity between each text vector in the vector database and the retrieval semantic vector; and acquire at least one target text vector from the vector database based on the similarity.

[0075] In this embodiment of the application, the second acquisition module is used to acquire the target document identifier and the target sequence identifier associated with the target text vector; query the corresponding target text slice from the slice relationship table using the target document identifier and the target sequence identifier; query the preceding slice sequence and the following slice sequence corresponding to the target text slice in the slice relationship table; extract a first number of preceding text slices from the preceding slice sequence in reverse order, and extract a second number of following text slices from the following slice sequence in forward order.

[0076] In this embodiment of the application, the apparatus further includes: a determining module, configured to receive a first quantity specified by the user for a preceding slice and a second quantity specified for a subsequent slice; determine the required number of segments based on the semantic integrity of the target text slice, and calculate the first quantity for the preceding slice and the second quantity for the subsequent slice based on the required number of segments; and calculate the first quantity for the preceding slice and the second quantity for the subsequent slice based on the document length of the original document content and a preset ratio parameter.

[0077] In this embodiment of the application, the construction module 43 is specifically used to perform deduplication processing on the preceding text slice, the target text slice, and the subsequent text slice to obtain deduplicated text slices; sort the deduplicated text slices according to the sequence identifier to generate continuous slice content; and reconstruct the document structure based on the original segmentation position of the continuous slice content to generate restored document content.

[0078] In this embodiment of the application, the construction module 43 is specifically used to identify the semantic coherence of the original segmentation position in the continuous slice content; and to add structured elements to the continuous slice content based on the semantic coherence to generate the restored document content.

[0079] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 5 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system).

[0080] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0081] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0082] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device as shown by a landing page for an app. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, which can be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0083] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0084] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0085] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0086] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A text processing method, characterized in that, The method includes: At least one target text vector is retrieved from a vector database according to a query request, wherein the vector database includes multiple text vectors, and the text vectors are obtained based on text slicing; Obtain the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice; The document content is reconstructed based on the target text slice, the preceding text slice, and the following text slice.

2. The method according to claim 1, characterized in that, Before retrieving multiple text vectors from the vector database according to the query request, the method further includes: Retrieve the original document content and its corresponding document identifier; The original document content is segmented according to preset segmentation rules to obtain multiple text slices; Based on the segmentation order corresponding to multiple text slices, generate an order identifier associated with each text slice, and construct a corresponding text vector based on the semantic features of the text slices; Each text slice is associated with the sequence identifier and the document identifier and stored in the slice relationship table, and the text vector corresponding to each text slice is associated with the sequence identifier and the document identifier and stored in the vector database.

3. The method according to claim 1, characterized in that, The step of retrieving at least one target text vector from the vector database according to the query request includes: Parse the retrieval semantic vector in the query request; Calculate the similarity between each text vector in the vector database and the retrieved semantic vector; At least one target text vector is obtained from the vector database based on the similarity.

4. The method according to claim 2, characterized in that, The step of obtaining the target text slice corresponding to the target text vector, as well as the preceding and following text slices adjacent to the target text slice, includes: Obtain the target document identifier and target sequence identifier associated with the target text vector; The corresponding target text slice is retrieved from the slice relationship table using the target document identifier and the target sequence identifier; Query the preceding and following slice sequences corresponding to the target text slice in the slice relationship table; A first number of preceding text slices are extracted from the preceding slice sequence in reverse order, and a second number of following text slices are extracted from the following slice sequence in forward order.

5. The method according to claim 4, characterized in that, The first quantity and / or the second quantity are determined based on at least one of the following methods: Receives a first quantity specified by the user for the preceding slice and a second quantity specified for the subsequent slice; The required number of segments is determined based on the semantic completeness of the target text slice, and a first number for the preceding slice and a second number for the subsequent slice are calculated based on the required number of segments. The first number of slices and the second number of slices are calculated based on the document length of the original document content and preset ratio parameters.

6. The method according to claim 1, characterized in that, The process of constructing and restoring document content based on the target text slice, the preceding text slice, and the following text slice includes: The preceding text slice, the target text slice, and the following text slice are deduplicated to obtain a deduplicated text slice. The deduplicated text slices are sorted according to the sequence identifier to generate continuous slice content; The document structure is reconstructed based on the original segmentation positions of the continuous slice content, and the restored document content is generated.

7. The method according to claim 6, characterized in that, The process of reconstructing the document structure based on the original segmentation positions of the continuous slice content to generate the restored document content includes: Identify the semantic coherence of the original segmentation positions within the continuous slice content; Based on the semantic coherence, structured elements are added to the continuous slice content to generate the restored document content.

8. A text processing device, characterized in that, The device includes: The first acquisition module is used to acquire at least one target text vector from a vector database according to a query request, wherein the vector database includes multiple text vectors, and the text vectors are obtained based on text slicing; The second acquisition module is used to acquire the target text slice corresponding to the target text vector, as well as the preceding text slice and the following text slice adjacent to the target text slice; A construction module is used to construct and restore document content based on the target text slice, the preceding text slice, and the following text slice.

9. A computer device, characterized in that, include: A memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, the processor executing the computer instructions to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method of any one of claims 1 to 7.

Images