Text data chunking processing methods and related equipment

Abstract

This application provides a text data segmentation processing method and related equipment, belonging to the fields of natural language processing and knowledge graph technology. The method includes: reading text segment data from a document to be processed, then preprocessing the text segment data to obtain a first sentence sequence; dynamically determining a second sentence sequence from the first sentence sequence using a streaming sliding buffer processing mechanism; semantically encoding the second sentence sequence using a preset encoder, then performing semantic coherence analysis based on the obtained semantic vector data to obtain expected semantic breakpoint data; and segmenting the text data based on the expected semantic breakpoint data to obtain a text data segmentation result. This application can achieve dynamic recognition of text semantic boundaries, effectively improving the semantic consistency and structural integrity of text segments, and effectively supporting large-scale text segmentation processing.

Description

Technical Field

[0001] This application relates to the fields of natural language processing and knowledge graph technology, and in particular to a method and related equipment for text data segmentation processing. Background Technology

[0002] In knowledge graph construction and intelligent diagnostic systems, text segmentation is a key step in preprocessing long documents and extracting structured knowledge. However, related technologies often use a mechanical division method for text segmentation, which can easily disrupt the semantic coherence of documents, fail to dynamically adapt to semantic boundaries, result in low-quality text segmentation, and make it difficult to process large-scale documents.

[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention

[0004] The main objective of this application is to propose a text data segmentation processing method and related device, which can realize dynamic recognition of text semantic boundaries, effectively improve the semantic consistency and structural integrity of text segments, and effectively support text segmentation processing of large-scale texts.

[0005] To achieve the above objectives, one aspect of this application proposes a text data chunking processing method, the method comprising: The text paragraph data is read from the document to be processed, and then the text paragraph data is preprocessed to obtain the first sentence sequence; The second sentence sequence is dynamically determined from the first sentence sequence using a streaming sliding buffer processing mechanism; The second sentence sequence is semantically encoded by a preset encoder, and then semantic coherence analysis is performed based on the obtained semantic vector data to obtain the expected semantic breakpoint data. The text data is segmented based on the expected semantic breakpoint data to obtain the text data segmentation result.

[0006] In some embodiments, the step of reading text paragraph data from the document to be processed, and then preprocessing the text paragraph data to obtain a first sentence sequence includes: The document to be processed is read using a preset streaming reading strategy to obtain the text paragraph data; wherein, the preset streaming reading strategy includes a batch streaming reading strategy with a buffer. Sentence segmentation is performed on the text paragraph data to obtain the first sentence sequence; Each sentence in the first sentence sequence is tokenized to obtain a token length sequence.

[0007] In some embodiments, dynamically determining the second sentence sequence from the first sentence sequence using a streaming sliding buffer processing mechanism includes: Construct a buffer queue based on preset capacity parameters; The expected sentence interval to be loaded is dynamically calculated based on the preset processing pointer and the preset capacity parameter, and then the second sentence sequence is determined from the first sentence sequence based on the expected sentence interval to be loaded. The second sentence sequence is loaded into the buffer queue, and the preset processing pointer is updated according to the second sentence sequence.

[0008] In some embodiments, the step of semantically encoding the second sentence sequence using a preset encoder, and then performing semantic coherence analysis based on the obtained semantic vector data to obtain desired semantic breakpoint data includes: The semantic vectors of all adjacent sentence pairs in the second sentence sequence are input into the preset encoder to calculate the semantic vectors of the sentence pairs; wherein, the preset encoder includes the Siamese Sentence-BERT encoder; Based on the semantic vectors of the sentences, cosine similarity is calculated to obtain sentence pair similarity data; Based on the sentence similarity data, semantic jump data is calculated, and then candidate breakpoint data is determined through the semantic jump data; Based on the candidate breakpoint data, semantic breakpoint analysis is performed using preset length constraints to obtain the desired semantic breakpoint data; wherein, the preset length constraints are determined by a preset maximum token length parameter.

[0009] In some embodiments, before performing semantic vector calculation on all adjacent sentence pairs in the second sentence sequence input to the preset encoder to obtain sentence pair semantic vectors, the method further includes: Construct a pre-defined Sentence-BERT network model based on the Transformer architecture; The preset Sentence-BERT network model is used as a sub-network to deploy a Siamese network, thereby constructing the Siamese Sentence-BERT encoder; wherein, each sub-network in the Siamese Sentence-BERT encoder shares the same parameter set.

[0010] In some embodiments, the step of segmenting text data based on the expected semantic breakpoint data to obtain text data segmentation results includes: The second sentence sequence is segmented based on the expected semantic breakpoint data to generate target text blocks; The target text block is semantically encoded to obtain a target semantic vector; A mapping relationship is established between the target text block and the target semantic vector, and the mapping relationship is stored in a preset structured storage unit to obtain the text data block result.

[0011] To achieve the above objectives, another aspect of this application provides a text data chunking processing apparatus, the apparatus comprising: The first module is used to read text paragraph data from the document to be processed, and then preprocess the text paragraph data to obtain the first sentence sequence; The second module is used to dynamically determine the second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism; The third module is used to perform semantic encoding on the second sentence sequence through a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain the expected semantic breakpoint data. The fourth module is used to segment the text data into blocks based on the expected semantic breakpoint data, and obtain the text data segmentation results.

[0012] To achieve the above objectives, another aspect of this application provides an electronic device, the electronic device comprising: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0013] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0014] To achieve the above objectives, another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method. The embodiments of this application include at least the following beneficial effects: This application provides a text data chunking processing method, apparatus, electronic device, storage medium, and program product. This solution reads text segment data from a document to be processed, preprocesses the text segment data to obtain a first sentence sequence, and dynamically determines a second sentence sequence from the first sentence sequence using a streaming sliding buffer processing mechanism. Then, the embodiments of this invention perform semantic encoding on the second sentence sequence using a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain expected semantic breakpoint data. Based on the expected semantic breakpoint data, text data is chunked to obtain the text data chunking result, thus realizing chunking processing of text data. It is easily understood that by performing semantic coherence analysis on the semantic vector data to determine the corresponding semantic breakpoint positions, i.e., the expected semantic breakpoint data, the embodiments of this invention can achieve dynamic identification of text semantic boundaries and effectively improve the semantic consistency and structural integrity of text chunking. Simultaneously, by employing a streaming sliding buffer mechanism to determine the second sentence sequence from the first sentence sequence, streaming processing can be achieved while reading and chunking, alleviating the problem of excessive memory consumption when processing large-scale documents, and effectively supporting large-scale text chunking processing. Attached Figure Description

[0015] Figure 1 This is a flowchart of the text data segmentation processing method provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the structure of the text data chunking processing device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0018] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Before providing a detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained first. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] BERT model: It is a pre-trained language model based on the Transformer architecture. Through bidirectional context modeling and large-scale unsupervised pre-training, it significantly improves the depth and accuracy of text understanding and drives performance breakthroughs in many NLP tasks.

[0022] In knowledge graph construction and intelligent diagnostic systems, text segmentation is a key step in preprocessing long documents and extracting structured knowledge. However, related technologies often use a mechanical division method for text segmentation, which can easily disrupt the semantic coherence of documents, fail to dynamically adapt to semantic boundaries, result in low-quality text segmentation, and make it difficult to process large-scale documents.

[0023] For example, traditional text segmentation methods mainly include fixed-size segmentation, sliding window segmentation, and paragraph-based segmentation. Fixed-size segmentation mechanically divides text according to a preset number of tokens, ignoring the semantic coherence between sentences. This easily disrupts key diagnostic logic chains such as "fault phenomenon—failure mechanism—system response," leading to semantic gaps and inference biases in subsequent retrieval and alignment stages. While sliding window segmentation introduces overlapping areas to alleviate contextual fragmentation, it still cannot fundamentally identify and maintain the integrity of semantic units. Furthermore, paragraph-based segmentation relies on document layout and punctuation structure, and its performance is limited in professional texts with irregular formatting or logical cross-paragraph structures. Therefore, traditional text segmentation methods have the following common limitations: 1. They cannot dynamically perceive the semantic coherence within the text, resulting in poor performance in terms of logical chain preservation, title-content consistency, and technical entity integrity; 2. When processing large-scale documents, the entire text usually needs to be loaded at once, which consumes a lot of memory and is difficult to handle the processing needs of large files such as those exceeding 300MB; 3. They lack streaming processing capabilities and cannot support scenarios where segmentation is performed while reading and semantic changes are adapted in real time, resulting in insufficient flexibility and scalability.

[0024] In view of this, embodiments of this application provide a text data segmentation processing method, apparatus, electronic device, storage medium, and program product. This scheme obtains text segment data by reading the document to be processed, then preprocesses the text segment data to obtain a first sentence sequence, and dynamically determines a second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism. Then, embodiments of this invention perform semantic encoding on the second sentence sequence using a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain expected semantic breakpoint data. Based on the expected semantic breakpoint data, text data is segmented to obtain text data segmentation results, realizing segmentation processing of text data. This enables dynamic identification of text semantic boundaries, effectively improving the semantic consistency and structural integrity of text segments, and effectively supporting large-scale text segmentation processing.

[0025] The text data segmentation processing method provided in this application relates to the fields of natural language processing and knowledge graph technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the text data segmentation processing method, but is not limited to the above forms.

[0026] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0027] Figure 1 This is an optional flowchart of the text data segmentation processing method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S110 to S140.

[0028] Step S110: Read the text paragraph data from the document to be processed, and then preprocess the text paragraph data to obtain the first sentence sequence.

[0029] Step S120: Dynamically determine the second sentence sequence from the first sentence sequence using a streaming sliding buffer processing mechanism.

[0030] Step S130: Semantically encode the second sentence sequence using a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain the expected semantic breakpoint data.

[0031] Step S140: Segment the text data according to the expected semantic breakpoint data to obtain the text data segmentation result.

[0032] In this specific embodiment, the present invention first reads text paragraph data from the document to be processed and preprocesses the text paragraph data to obtain a first sentence sequence. Specifically, the document to be processed in this embodiment refers to a document that requires text data block processing, such as a maintenance manual, a fault case library, or other text sources. Accordingly, the present invention reads document content from the document to be processed and preprocesses the read document content, such as by parsing it step by step, thereby identifying and removing irrelevant non-subject text content, and extracting continuous plain text paragraphs, i.e., the first sentence sequence. Next, the present invention dynamically determines a second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism. Specifically, in order to enable the system to process the first sentence sequence continuously and efficiently with a constant and low memory usage, so as to facilitate subsequent adaptive text data block processing, the present invention uses a streaming sliding buffer processing mechanism to dynamically select the target sentences for block processing from the first sentence sequence, i.e., to determine the second sentence sequence. In this embodiment, the streaming sliding buffer processing mechanism determines the number of sentences processed each time through a preset capacity window parameter and dynamically determines the target sentence sequence through a preset processing pointer, thereby realizing streaming progressive processing. Furthermore, in this embodiment of the invention, the second sentence sequence is semantically encoded using a preset encoder, and then semantic coherence analysis is performed based on the obtained semantic vector data to obtain expected semantic breakpoint data. Specifically, in this embodiment of the invention, the preset encoder refers to a semantic encoder, such as a BERT encoder. Correspondingly, this embodiment of the invention inputs the second sentence sequence into the preset encoder for semantic information calculation, and performs semantic coherence analysis through a similarity metric mechanism, that is, determines whether the semantics between sentences are coherent, thereby determining the expected semantic breakpoint data. Here, the expected semantic breakpoint data refers to the position where semantics jump, that is, the position where the semantics between preceding and following sentences are not coherent. Finally, this embodiment of the invention performs text data segmentation based on the expected semantic breakpoint data to obtain text data segmentation results. Specifically, this embodiment of the invention performs text segmentation processing on the second sentence sequence according to the result of semantic coherence analysis, that is, the expected semantic breakpoint data, to obtain text data segmentation results. In this embodiment of the invention, a streaming activity buffer mechanism is used to dynamically select a second sentence sequence from a first sentence sequence. Then, a preset encoder is used to perform semantic encoding and semantic coherence analysis on the second sentence sequence. After the second sentence sequence is segmented into text blocks according to the determined semantic breakpoints, the process returns to the step of selecting a second sentence sequence from the first sentence sequence. This achieves streaming cyclical processing until all sentences in the first sentence sequence have been processed, thereby obtaining the final text data segmentation result.

[0033] In some embodiments of the present invention, text paragraph data is read from the document to be processed, and then the text paragraph data is preprocessed to obtain a first sentence sequence, including but not limited to the following steps: The document to be processed is read using a preset streaming strategy to obtain text paragraph data. The preset streaming strategy includes a buffered batch streaming strategy.

[0034] Sentence segmentation is performed on the text paragraph data to obtain the first sentence sequence.

[0035] Each sentence in the first sentence sequence is tokenized to obtain a token length sequence.

[0036] In this specific embodiment, the present invention first reads the content of the document to be processed using a preset streaming reading strategy to obtain text segment data. Then, it performs sentence segmentation processing based on the text segment data to obtain a first sentence sequence. Each sentence in the first sentence sequence is then tokenized to obtain a token length sequence. Specifically, since loading a large file at once may cause memory crashes, the present invention employs a preset streaming reading strategy to read the document to be processed to mitigate this problem. Specifically, the preset streaming reading strategy in this embodiment includes a buffered batch streaming reading strategy. The present invention reads the document content from the document to be processed in batches, reading a preset-sized text segment each time, parsing it step by step, identifying and removing irrelevant non-subject text content, thereby extracting continuous plain text segments, which can alleviate the memory crash problem caused by loading a very large file at once.

[0037] For example, embodiments of the present invention employ a buffered batch streaming strategy to process input documents. The total document size is... The embodiments of the present invention preset the batch size for reading. Initialize the read pointer and create a text buffer. Accordingly, when At that time, embodiments of the present invention read the next block of bytes from the document. The size is shown in the following formula:

[0038] Next, the byte block is decoded into a text string. And append to the buffer:

[0039] At the same time, update the read position:

[0040] Then, the function is called in this embodiment of the invention. Scan the buffer to locate the nearest reliable sentence end boundary. This involves extracting the complete sentence sequence from the buffer. The remaining incomplete text is retained in middle:

[0041] This process is repeated until no complete sentences can be extracted from the buffer. Then, embodiments of the invention process the extracted text paragraphs. Sentence segmentation is performed, and the token length for each sentence is calculated. Sentence boundary recognition is based on a combination of rules and models: [The text abruptly ends here, likely due to an incomplete sentence or a formatting error.] A hybrid sentence segmentation strategy is applied to output an atomic sentence sequence. In this embodiment of the invention, a tokenizer matching a preset encoder is used for each sentence. Tokenize and calculate the number of tokens. Forming pairs of sentences and their lengths Accordingly, the sentence sequence generated in this batch is appended to the global sentence sequence. In the middle, update the corresponding length information:

[0042] Among them, when And buffer When the string is empty (or only fragments remain that cannot form a complete sentence), the first sentence sequence reading and processing step ends, and the final output of this step is an ordered sentence sequence. and its corresponding token length sequence .

[0043] In some embodiments of the present invention, a second sentence sequence is dynamically determined from a first sentence sequence using a streaming sliding buffer processing mechanism, including but not limited to the following steps: Build a buffer queue based on preset capacity parameters.

[0044] The expected sentence range is dynamically calculated based on the preset processing pointer and preset capacity parameters, and then the second sentence sequence is determined from the first sentence sequence based on the expected sentence range.

[0045] The second sentence sequence is loaded into the buffer queue, and the preset processing pointer is updated according to the second sentence sequence.

[0046] In this specific embodiment, the present invention first constructs a buffer queue according to a preset capacity parameter, and dynamically calculates the expected sentence interval to be loaded based on a preset processing pointer and the preset capacity parameter. Based on the expected sentence interval, a second sentence sequence is determined from the first sentence sequence, and then loaded into the buffer queue to facilitate subsequent block processing. The preset processing pointer is updated according to the second sentence sequence to achieve streaming processing. Specifically, the present invention first initializes two key state variables: the preset processing pointer and the preset capacity parameter of the buffer queue. The preset capacity parameter refers to the upper limit of the buffer queue's capacity. The present invention sets a global processing pointer. That is, the preset processing pointer, whose initial value is defined as This indicates that processing should begin from the first sentence of the sequence. Simultaneously, an empty buffer queue is created. And set a fixed capacity limit for it. This capacity It is a preset constant that satisfies (Much smaller than the total number of sentences in the document), this design is key to controlling memory consumption. Specifically, in each processing loop, the system adjusts the current pointer... Dynamically calculate the sentence range to be loaded this time and define the end position of loading. for:

[0047] Where, in the formula This represents the total number of sentences in the document. Then, in this embodiment of the invention, the original sentence sequence... The corresponding subsequence is loaded into the buffer queue as shown in the following formula:

[0048] This invention embodiment enables the memory to be maintained only at any given time through this operation. The data for each sentence allows for strict control over memory usage.

[0049] Furthermore, in the buffer queue Once loaded, the data is passed to subsequent steps for semantic computation and chunking decisions. For example, after processing, a new text chunk is partitioned from the current queue. This block ends at sentence (in Subsequently, this embodiment of the invention performs a core pointer update operation to advance the streaming processing, as shown in the following equation:

[0050] In this embodiment of the invention, the global pointer is updated to the starting position of the next round of processing using this formula. Afterwards, the current queue is cleared. And based on the new pointer The data loading process described above is repeated, thus forming a closed-loop streaming cycle of "loading-processing-updating". This embodiment of the invention uses this mechanism to transform the processing of massive documents into iterative operations on a fixed-size data window, with minimal memory consumption. It is unrelated to the total document size, only related to the preset capacity. Proportional ( This effectively alleviates the memory bottleneck of traditional methods.

[0051] In some embodiments of the present invention, the second sentence sequence is semantically encoded using a preset encoder, and then semantic coherence analysis is performed based on the obtained semantic vector data to obtain desired semantic breakpoint data, including but not limited to the following steps: The semantic vectors of sentence pairs are calculated by inputting all adjacent sentence pairs in the second sentence sequence into the preset encoder. The preset encoder includes the Siamese Sentence-BERT encoder.

[0052] The sentence similarity data is obtained by calculating the cosine similarity of the semantic vectors of the sentences.

[0053] Semantic jump data is calculated based on sentence similarity data, and then candidate breakpoint data is determined through semantic jump data.

[0054] Semantic breakpoint analysis is performed on the candidate breakpoint data using preset length constraints to obtain the desired semantic breakpoint data. The preset length constraints are determined by a preset maximum token length parameter.

[0055] In this specific embodiment, the present invention first inputs each adjacent sentence pair in the second sentence sequence into a preset encoder for semantic vector calculation, and then calculates cosine similarity data based on the obtained semantic vectors of the sentence pairs to obtain sentence pair similarity data. Specifically, the preset encoder in this embodiment includes a Siamese Sentence-BERT encoder. The present invention calculates semantic vectors for the input sentence pairs using the Siamese Sentence-BERT encoder and calculates the corresponding cosine similarity to obtain sentence pair similarity data. Here, a sentence pair in this embodiment refers to a pair of two adjacent sentences in the second sentence sequence. By analyzing the similarity of each sentence pair, the coherence changes between adjacent sentences are determined, thereby locating the position where a significant semantic jump occurs. Correspondingly, in the streaming buffer queue Q, the present invention performs dynamic and adaptive block-segmentation decisions based on the semantic information calculated in real time by the Siamese Sentence-BERT encoder. Its core objective is to maximize the semantic cohesion of text blocks, ensuring high semantic coherence of sentences within the same text block. This objective can be defined by a quantification formula:

[0056] Where, in the formula Represents two sentence vectors and The cosine similarity between them. Accordingly, in the embodiments of the present invention, subsequent block-segmentation decisions are all centered around this objective, maximizing the semantic cohesion of text blocks while satisfying length constraints.

[0057] For example, embodiments of the present invention for loading into the current buffer queue Each sentence in Invoke the initialized twin Sentence-BERT encoder The fixed-length semantic vector representation is calculated in real time, as shown in the following formula:

[0058] Where, in the formula These are shared parameters for the encoder. This embodiment of the invention uses this step to convert text sentences in the queue into semantic vectors that can be mathematically compared. Next, based on the calculated sentence vectors, this embodiment of the invention quantifies the semantic coherence changes between adjacent sentences to locate positions where significant semantic jumps occur. Specifically, the cosine similarity of adjacent sentence pairs is first calculated: for all adjacent sentence pairs in the queue... The cosine similarity between their sentence vectors is calculated using the following formula:

[0059] Where, in the formula Represents the vector dot product. This represents the L2 norm of a vector.

[0060] Furthermore, in this embodiment of the invention, semantic jump degree data is calculated based on sentence similarity data, and then candidate breakpoint data is determined through semantic jump degree data. Semantic breakpoint analysis is then performed on the candidate breakpoint data using preset length constraints to obtain the desired semantic breakpoint data. Specifically, in this embodiment of the invention, semantic jump degree is defined based on cosine similarity to transform cosine similarity into a more intuitive semantic jump degree, as shown in the following formula:

[0061] Where, in the formula The range of its value is [0,2]. The larger the value, the more accurate the sentence is. arrive The lower the semantic coherence, the more likely that position is an ideal semantic block boundary.

[0062] Next, after quantifying all potential breakpoints, i.e., determining the candidate breakpoint data, this embodiment of the invention needs to make a final segmentation decision under a hard length constraint. Accordingly, this embodiment of the invention first sets the starting sentence index of the current block to be constructed as... Based on the preset maximum token length per block. To clearly define the length limit of a single text block, this embodiment of the invention constructs a hard length constraint formula, that is, a preset length constraint condition: for any generated text block The total number of tokens in all sentences contained therein must strictly satisfy the following condition. ,in The token count represents sentence s. Based on this constraint, find the index of the furthest sentence in the buffer that satisfies the length summation constraint. :

[0063] Then, the embodiment of the present invention locates the optimal semantic breakpoint: within the scalable range Within, find the position with the greatest semantic leap. Its expression is shown in the following formula:

[0064] Accordingly, when When this occurs, it indicates the existence of a selectable semantic breakpoint space. In this case, the segmentation position is taken as the position with the largest local semantic jump, i.e. ,exist and Divide into between. And when When this occurs, it indicates that the scalability range is extremely small under the length constraint, and there is insufficient space to select semantic breakpoints. Therefore, the block is directly extended to the maximum allowed length, and the split position is set to [missing information]. ,exist The segmentation is completed at that point. Or, if... Then in and The blocks are divided into sections, and the current block contains sentences. to Or, if If there is no identifiable breakpoint, then extend the block to the maximum permissible length, i.e., at... The current block is divided into segments, and the current block contains sentences. to The embodiments of the present invention determine the final breakpoint location, i.e., the expected semantic breakpoint data, through the above semantic breakpoint analysis.

[0065] In some embodiments of the present invention, before performing semantic vector calculation on all adjacent sentences in the second sentence sequence input to a preset encoder to obtain sentence-to-sentence semantic vectors, the text data segmentation processing method provided in the embodiments of the present invention further includes, but is not limited to, the following steps: A pre-defined Sentence-BERT network model is built based on the Transformer architecture.

[0066] A pre-defined Sentence-BERT network model is deployed as a sub-network in a Siamese network to construct a Siamese Sentence-BERT encoder. Each sub-network in the Siamese Sentence-BERT encoder shares the same parameter set.

[0067] In this specific example, this embodiment of the invention constructs a pre-defined Sentence-BERT network model using the Transformer architecture, and then deploys it as a sub-network in a Siamese network to obtain a Siamese Sentence-BERT encoder. Specifically, this embodiment of the invention uses a Sentence-BERT model based on the Transformer architecture and deploys it as a Siamese network. Accordingly, in this embodiment of the invention, the two sub-networks share the same parameter set. So that for any two sentences and The encoding process is consistent. Its core encoding function is defined as follows: For an input sentence In this embodiment of the invention, the context representation of each token is first obtained through a shared encoder. Depend on Composed of several tokens, the last hidden state sequence output by the encoder is: ,in Sentence vector The calculation is performed using mean pooling, as shown in the following formula:

[0068] Where, in the formula As a shared parameter set for the twin network, since the embodiments of the present invention adopt a weight-sharing twin BERT architecture, the two sub-networks share the same set of parameters. This ensures consistency in the encoding process for any sentence. This is the last hidden state output by the Transformer encoder. For sentences The number of tokens included, through this formula, can convert a sentence into a fixed-dimensional semantic vector, providing a unified and comparable basis for subsequent semantic similarity calculations. As a sentence The fixed-dimensional semantic representation. Furthermore, embodiments of the present invention use cosine similarity to measure the semantic similarity between two sentences. and Semantic coherence, cosine similarity The calculation formula is shown below:

[0069] in, Represents the vector dot product. This represents the L2 norm of a vector. In streaming block decision-making, it is necessary to quantify the semantic changes between adjacent sentences. Accordingly, embodiments of the present invention determine the semantic jump degree based on cosine similarity. A larger value indicates lower semantic coherence between sentences, and is more likely to be a potential semantic breakpoint. The encoder constructed as described above is used in this embodiment of the invention. The above formula can map any sentence into a comparable semantic vector, providing a computational basis for dynamically evaluating semantic boundaries in streaming processing.

[0070] In some embodiments of the present invention, text data is segmented based on expected semantic breakpoint data to obtain text data segmentation results, including but not limited to the following steps: The second sentence sequence is segmented based on the expected semantic breakpoint data to generate target text blocks.

[0071] Semantic encoding is performed on the target text block to obtain the target semantic vector.

[0072] A mapping relationship is established between the target text block and the target semantic vector, and the data is stored in a preset structured storage unit to obtain the text data block result.

[0073] In this specific embodiment, the present invention first segments the second sentence sequence based on the expected semantic breakpoint data to generate target text blocks, and then performs semantic encoding on the target text blocks to obtain target semantic vectors. Subsequently, a mapping relationship is established between the target text blocks and the target semantic vectors, and the results are stored in a preset structured storage unit to obtain the text data segmentation result. Specifically, the present invention will generate a segmentation result containing text blocks from sentences... arrive or The latest text block of a continuous sequence The generated text is immediately converted into a semantic vector and stored in the final result set. Accordingly, this embodiment of the invention encodes the generated target text block by calling the corresponding encoder to generate its corresponding semantic vector representation, for example, by calling the previously constructed Sentence-BERT encoder. For the entire text block Encode the corresponding semantic vector representation, as shown in the following formula:

[0074] In this embodiment of the invention, the encoding method can be a weighted average of all sentence vectors within a block or directly using [CLS] Token representation. Then, this embodiment of the invention will generate semantic vectors... Its corresponding original text block Establish the mapping relationship and store it in the structured storage system: Storage←( , ) Finally, this embodiment of the invention updates the result set, that is, it simultaneously updates the text blocks. Add to the final block result set middle:

[0075] It should be noted that, after completing the storage of the current block, this embodiment of the invention updates the global processing pointer according to the determined segmentation point. To advance the streaming processing process:

[0076] Then, a loop termination check is performed. This closed loop is driven by pointer p and continues until all sentence sequences S have been processed. For example, if If the condition is met, it indicates that there are still unprocessed sentences. In this embodiment of the invention, the process jumps to the step of dynamically determining the second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism to load the next batch of sentence data and continue the loop; or, if If the condition is met, it indicates that all sentences have been processed and the loop terminates. Accordingly, when the loop termination condition is met, this embodiment of the invention outputs the stored set of vector segments and their corresponding text block indices:

[0077] Where, in the formula It is a set of semantic vectors for all text blocks, which can be directly used for downstream tasks such as dense retrieval, knowledge alignment and large model question answering; This is the corresponding set of original text blocks, used for backtracking or interpretation when necessary.

[0078] It is readily understood that embodiments of the present invention utilize Sentences trained on Siamese networks. The BERT model calculates the semantic relevance between sentences in real time and implements streaming processing by reading and segmenting text simultaneously through a sliding buffer mechanism. This ensures that logical units remain intact within the same text block while meeting length constraints, such as logical units like "abnormal behavior - fault mechanism - system behavior," achieving high-quality segmentation of large-scale technical documents with semantic consistency and structural integrity. Specifically, this embodiment of the invention achieves real-time dynamic recognition of text semantic boundaries through an adaptive streaming segmentation mechanism. During streaming reading, this embodiment continuously senses the semantic coherence between sentences and adaptively segments text blocks to ensure that diagnostic logical units maintain semantic integrity. This process is completed entirely within the streaming processing framework, without the need to preload the complete document, effectively supporting real-time sentence-by-sentence processing of large-scale texts. The adaptive semantic segments generated in this embodiment significantly improve the accuracy and contextual consistency of knowledge retrieval, providing an efficient text preprocessing solution for diagnostic knowledge graph construction. It also possesses good memory friendliness and adaptability, and the generated semantically cohesive text blocks significantly improve the accuracy and contextual consistency of dense retrieval, making it suitable for real-time preprocessing scenarios of professional long documents, such as repair manuals and fault case libraries.

[0079] Please see Figure 2 This application also provides a text data chunking processing apparatus that can implement the above-described method. The apparatus includes: The first module 210 is used to read text paragraph data from the document to be processed, and then preprocess the text paragraph data to obtain the first sentence sequence.

[0080] The second module 220 is used to dynamically determine the second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism.

[0081] The third module 230 is used to perform semantic encoding on the second sentence sequence through a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain the expected semantic breakpoint data.

[0082] The fourth module 240 is used to segment text data into blocks based on the expected semantic breakpoint data, and obtain the text data segmentation results.

[0083] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0084] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0085] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0086] Please see Figure 3 , Figure 3 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 310 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 320 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 320 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310 using the methods described in the embodiments of this application. Input / output interface 330 is used to realize information input and output; The communication interface 340 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 350 transmits information between various components of the device (e.g., processor 310, memory 320, input / output interface 330, and communication interface 340); The processor 310, memory 320, input / output interface 330 and communication interface 340 are connected to each other within the device via bus 350.

[0087] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0088] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0089] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0090] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0091] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0092] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0093] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0096] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0097] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0099] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0100] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0101] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0102] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for processing text data in blocks, characterized in that, The method includes the following steps: The text paragraph data is read from the document to be processed, and then the text paragraph data is preprocessed to obtain the first sentence sequence; The second sentence sequence is dynamically determined from the first sentence sequence using a streaming sliding buffer processing mechanism; The second sentence sequence is semantically encoded by a preset encoder, and then semantic coherence analysis is performed based on the obtained semantic vector data to obtain the expected semantic breakpoint data. The text data is segmented based on the expected semantic breakpoint data to obtain the text data segmentation result.

2. The method according to claim 1, characterized in that, The process of reading text paragraph data from the document to be processed, and then preprocessing the text paragraph data to obtain a first sentence sequence includes: The document to be processed is read using a preset streaming reading strategy to obtain the text paragraph data; wherein, the preset streaming reading strategy includes a batch streaming reading strategy with a buffer. Sentence segmentation is performed on the text paragraph data to obtain the first sentence sequence; Each sentence in the first sentence sequence is tokenized to obtain a token length sequence.

3. The method according to claim 1, characterized in that, The step of dynamically determining the second sentence sequence from the first sentence sequence using a streaming sliding buffer processing mechanism includes: Construct a buffer queue based on preset capacity parameters; The expected sentence interval to be loaded is dynamically calculated based on the preset processing pointer and the preset capacity parameter, and then the second sentence sequence is determined from the first sentence sequence based on the expected sentence interval to be loaded. The second sentence sequence is loaded into the buffer queue, and the preset processing pointer is updated according to the second sentence sequence.

4. The method according to claim 1, characterized in that, The step of semantically encoding the second sentence sequence using a preset encoder, and then performing semantic coherence analysis based on the obtained semantic vector data to obtain expected semantic breakpoint data includes: The semantic vectors of all adjacent sentence pairs in the second sentence sequence are input into the preset encoder to calculate the semantic vectors of the sentence pairs; wherein, the preset encoder includes the Siamese Sentence-BERT encoder; Based on the semantic vectors of the sentences, cosine similarity is calculated to obtain sentence pair similarity data; Based on the sentence similarity data, semantic jump data is calculated, and then candidate breakpoint data is determined through the semantic jump data; Based on the candidate breakpoint data, semantic breakpoint analysis is performed using preset length constraints to obtain the desired semantic breakpoint data; wherein, the preset length constraints are determined by a preset maximum token length parameter.

5. The method according to claim 4, characterized in that, Before performing the step of inputting all adjacent sentence pairs in the second sentence sequence into the preset encoder to calculate semantic vectors and obtain sentence pair semantic vectors, the method further includes: Construct a pre-defined Sentence-BERT network model based on the Transformer architecture; The preset Sentence-BERT network model is used as a sub-network to deploy a Siamese network, thereby constructing the Siamese Sentence-BERT encoder; wherein, each sub-network in the Siamese Sentence-BERT encoder shares the same parameter set.

6. The method according to claim 4, characterized in that, The step of segmenting text data into blocks based on the expected semantic breakpoint data to obtain text data segmentation results includes: The second sentence sequence is segmented based on the expected semantic breakpoint data to generate target text blocks; The target text block is semantically encoded to obtain a target semantic vector; A mapping relationship is established between the target text block and the target semantic vector, and the mapping relationship is stored in a preset structured storage unit to obtain the text data block result.

7. A text data chunking processing device, characterized in that, The device includes: The first module is used to read text paragraph data from the document to be processed, and then preprocess the text paragraph data to obtain the first sentence sequence; The second module is used to dynamically determine the second sentence sequence from the first sentence sequence through a streaming sliding buffer processing mechanism; The third module is used to perform semantic encoding on the second sentence sequence through a preset encoder, and then perform semantic coherence analysis based on the obtained semantic vector data to obtain the expected semantic breakpoint data. The fourth module is used to segment the text data into blocks based on the expected semantic breakpoint data, and obtain the text data segmentation results.

8. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images