Delay block embedding processing method and device for long text and storage medium

Abstract

The invention discloses a delayed block embedding processing method and device for a long text and a storage medium, and belongs to the technical field of text processing. The method comprises the following steps: performing embedding processing on a target text to obtain an embedding vector sequence corresponding to the target text; according to a blocking strategy, based on the structure of the target text, determining blocking positions for blocking the target text and lengths of a plurality of text blocks obtained by blocking; and dividing the embedded vector sequence based on the block positions and the lengths of the plurality of text blocks, and carrying out average pooling on a plurality of sub-embedded vector sequences obtained after division to obtain embedded vectors corresponding to the plurality of text blocks. According to the embodiment of the invention, the semantic information of each text block can be better reserved through an embedding processing mode of delaying partitioning, and the problem of semantic loss caused by directly partitioning the text and then embedding in a traditional method is reduced, so that the accuracy of a model in a long text processing task is improved.

Description

Technical Field

[0001] This application belongs to the field of text processing technology, and in particular relates to a method, apparatus and storage medium for delayed block embedding of long text. Background Technology

[0002] In the field of natural language processing, the Transformer model has become one of the core technologies for processing text data. With its powerful parallel processing capabilities and ability to capture long-distance dependencies, the Transformer model is widely used in various tasks such as text generation, machine translation, and text classification. However, the Transformer model has a maximum context length limitation when processing long texts, making it difficult for the model to encode the entire text in one go.

[0003] In related technologies, long texts are typically pre-segmented into multiple shorter text blocks, and then each text block is encoded separately to generate an embedding vector. However, since the segmentation between text blocks is performed independently, the model ignores the semantic connections between adjacent text blocks during the encoding process, easily losing long-distance semantic dependencies across blocks, thus reducing the model's accuracy in long text processing tasks. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a delayed block embedding processing method, apparatus, and storage medium for long texts to improve the accuracy of models in long text processing tasks.

[0005] Firstly, this application provides a delayed chunking embedding method for long texts, including:

[0006] The target text is embedded to obtain the embedding vector sequence corresponding to the target text;

[0007] Based on the structure of the target text, the block positions and lengths of the resulting text blocks are determined according to the block segmentation strategy.

[0008] The embedding vector sequence is divided based on the block position and the length of the multiple text blocks, and the multiple sub-embedded vector sequences obtained after the division are averaged to obtain the embedding vectors corresponding to the multiple text blocks.

[0009] According to the delayed block embedding method for long texts in this application, the target text is embedded to obtain an embedding vector sequence corresponding to the target text; based on the structure of the target text, the block positions and lengths of the multiple text blocks obtained by the block division are determined according to the block division strategy; the embedding vector sequence is divided based on the block positions and the lengths of the multiple text blocks, and average pooling is performed on the multiple sub-embedded vector sequences obtained after the division to obtain the embedding vectors corresponding to the multiple text blocks. This embodiment of the application first performs overall embedding processing on the target text to obtain an overall embedding vector sequence, and then divides the embedding vector sequence according to the block division strategy. This delayed block embedding processing method can better preserve the semantic information of each text block, reducing the semantic loss problem caused by directly dividing the text into blocks and then embedding in traditional methods. Furthermore, through the average pooling operation, local features within the text blocks can be smoothly integrated, making the generated embedding vectors more accurately reflect the core semantics of the text blocks, thereby improving the accuracy of the model in long text processing tasks.

[0010] According to one embodiment of this application, the method further includes:

[0011] Obtain user problem information;

[0012] Based on the question information, target embedding vectors related to the question information are matched from a semantic vector library; the semantic vector library stores the embedding vectors corresponding to the plurality of text blocks;

[0013] The target embedding vector is converted into text content and combined with the question information to construct a prompt message;

[0014] The prompt information is input into the large language model to obtain the answer information corresponding to the question information output by the large language model.

[0015] In this embodiment, by storing the embedding vectors corresponding to multiple text blocks in the semantic vector library, and matching the target embedding vectors related to the question information from the semantic vector library, the content most relevant to the user's question can be quickly located. The target embedding vectors are converted into text content, which is used as information reference. Combined with the question information, prompt information is constructed. By utilizing the powerful language generation capabilities of the large language model, the answer information corresponding to the question information is obtained, thus achieving an accurate answer to the user's question.

[0016] According to one embodiment of this application, the embedding process of the target text to obtain the embedding vector sequence corresponding to the target text includes:

[0017] The target text is segmented using an embedding model to obtain a word sequence;

[0018] The word sequence is embedded to obtain the embedding vector sequence.

[0019] In this embodiment, the target text is segmented by an embedding model to obtain a word sequence. The word sequence is then embedded to map discrete words into a continuous vector space, so that the semantic information of the text is mapped into a numerical vector form that can be processed by a computer, thus better preserving the semantic information of the text.

[0020] According to one embodiment of this application, the embedding process of the lexical sequence to obtain the embedding vector sequence includes:

[0021] If the length of the lexical sequence is greater than the maximum processing length of the embedding model, a sliding window of target length is set;

[0022] The sliding window on the word sequence is slid with a target step size, and the word elements in the sliding window during the sliding process are embedded respectively; wherein the word elements in adjacent sliding windows have at least partial overlap.

[0023] In this embodiment, when the length of the lexical sequence exceeds the maximum processing length of the embedding model, a sliding window of a target length is set and slides on the lexical sequence with a target step size. This allows the long lexical sequence to be divided into multiple processable parts, and the lexical units in adjacent windows at least partially overlap. This reduces the problem of semantic loss caused by the disruption of semantic continuity between adjacent parts of the text during the division process. This enables the model to capture the semantic associations across windows when processing the lexical units in each sliding window, thereby improving the accurate representation of the overall semantics of the long text.

[0024] According to one embodiment of this application, the segmentation strategy includes segmentation based on the length of the text block, segmentation based on the number of sentences, segmentation based on punctuation marks, segmentation based on regular expressions, or segmentation based on paragraphs.

[0025] According to one embodiment of this application, the step of dividing the embedded vector sequence based on the block position and the length of the plurality of text blocks includes:

[0026] Obtain the embedding vector identifier corresponding to the block position in the embedding vector sequence;

[0027] The embedding vector sequence is divided according to the embedding vector identifier and the length of the plurality of text blocks.

[0028] In this embodiment, by obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence, the starting position of the block can be accurately located. The embedding vector sequence is divided by combining the length of multiple text blocks, so that each sub-embedded vector sequence corresponds to a text block. This division method not only ensures the consistency between the text block and the embedding vector sequence, but also makes full use of the text semantic information retained by the target text embedding process, so that the divided embedding vector sequence can accurately represent the semantic features of the corresponding text block.

[0029] According to one embodiment of this application, obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence includes:

[0030] When the segmentation strategy is based on punctuation or paragraph, the segmentation position is the sentence end punctuation mark or paragraph mark;

[0031] Determine the target embedding vector identifier corresponding to the sentence end punctuation mark or the paragraph mark in the embedding vector sequence.

[0032] In this embodiment, when the segmentation strategy is based on punctuation or paragraph, the sentence-ending punctuation mark or paragraph marker is determined as the segmentation position. This fully utilizes the structural characteristics of the text, ensuring that each segment remains relatively independent and complete semantically, and accurately locates the boundaries of the segments.

[0033] Secondly, this application provides a delayed chunking embedding processing apparatus for long texts, comprising:

[0034] The embedding module is used to embed the target text to obtain the embedding vector sequence corresponding to the target text;

[0035] The determination module is used to determine the block positions and lengths of the multiple text blocks obtained by the block division based on the structure of the target text according to the block division strategy.

[0036] The partitioning module is used to partition the embedding vector sequence based on the block position and the length of the multiple text blocks, and to perform average pooling on the multiple sub-embedded vector sequences obtained after partitioning to obtain the embedding vectors corresponding to the multiple text blocks.

[0037] According to the long text delayed segmentation embedding processing apparatus of this application, the target text is embedded to obtain an embedding vector sequence corresponding to the target text; the segmentation position and length of the multiple text blocks obtained by segmentation are determined based on the structure of the target text according to the segmentation strategy; the embedding vector sequence is divided based on the segmentation position and the length of the multiple text blocks, and the multiple sub-embedded vector sequences obtained after segmentation are averaged to obtain the embedding vectors corresponding to the multiple text blocks. This embodiment of the application first performs overall embedding processing on the target text to obtain an overall embedding vector sequence, and then divides the embedding vector sequence according to the segmentation strategy. This delayed segmentation embedding processing method can better preserve the semantic information of each text block, reducing the semantic loss problem caused by directly segmenting the text and then embedding it in traditional methods. Furthermore, through the average pooling operation, local features within the text block can be smoothly fused, so that the generated embedding vector can more accurately reflect the core semantics of the text block, thereby improving the accuracy of the model in long text processing tasks.

[0038] Thirdly, this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the delayed block embedding processing method for long text as described in the first aspect above.

[0039] Fourthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the delayed block embedding processing method for long text as described in the first aspect above.

[0040] Fifthly, this application provides a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the delayed block embedding processing method for long text as described in the first aspect above.

[0041] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the delayed block embedding processing method for long text as described in the first aspect above.

[0042] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects:

[0043] According to the delayed block embedding method for long texts in this application, the target text is embedded to obtain an embedding vector sequence corresponding to the target text; based on the structure of the target text, the block positions and lengths of the multiple text blocks obtained by the block division are determined according to the block division strategy; the embedding vector sequence is divided based on the block positions and the lengths of the multiple text blocks, and average pooling is performed on the multiple sub-embedded vector sequences obtained after the division to obtain the embedding vectors corresponding to the multiple text blocks. This embodiment of the application first performs overall embedding processing on the target text to obtain an overall embedding vector sequence, and then divides the embedding vector sequence according to the block division strategy. This delayed block embedding processing method can better preserve the semantic information of each text block, reducing the semantic loss problem caused by directly dividing the text into blocks and then embedding in traditional methods. Furthermore, through the average pooling operation, local features within the text blocks can be smoothly integrated, making the generated embedding vectors more accurately reflect the core semantics of the text blocks, thereby improving the accuracy of the model in long text processing tasks.

[0044] Furthermore, in some embodiments, by storing the embedding vectors corresponding to multiple text blocks in the semantic vector library, and matching the target embedding vectors related to the question information from the semantic vector library, the content most relevant to the user's question can be quickly located. The target embedding vectors are then converted into text content, which is used as an information reference. Combined with the question information, prompt information is constructed. By utilizing the powerful language generation capabilities of the large language model, the answer information corresponding to the question information is obtained, thus achieving an accurate answer to the user's question.

[0045] Furthermore, in some embodiments, the target text is segmented by an embedding model to obtain a word sequence. The word sequence is then embedded to map discrete words into a continuous vector space, so that the semantic information of the text is mapped into a numerical vector form that can be processed by a computer, thus better preserving the semantic information of the text.

[0046] Furthermore, in some embodiments, when the length of the lexical sequence exceeds the maximum processing length of the embedding model, a sliding window of a target length is set and slides on the lexical sequence with a target step size. This can divide the long lexical sequence into multiple processable parts, and the lexical units of adjacent windows at least partially overlap. This reduces the problem of semantic loss caused by the disruption of the semantic continuity of adjacent parts in the text during the segmentation process. It enables the model to capture the semantic associations across windows when processing the lexical units of each sliding window, thereby improving the accurate representation of the overall semantics of the long text.

[0047] Furthermore, in some embodiments, by obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence, the starting position of the block can be accurately located. The embedding vector sequence is divided in combination with the length of multiple text blocks, so that each sub-embedded vector sequence corresponds to a text block. This division method not only ensures the consistency between the text block and the embedding vector sequence, but also makes full use of the text semantic information retained by the target text embedding process, so that the divided embedding vector sequence can accurately represent the semantic features of the corresponding text block.

[0048] Furthermore, in some embodiments, when the segmentation strategy is based on punctuation or paragraph, the sentence-ending punctuation mark or paragraph marker is determined as the segmentation location, which makes full use of the structural characteristics of the text, so that each segment can remain relatively independent and complete in semantics, and the boundaries of the segments can be accurately located.

[0049] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0050] The above and / or additional aspects and advantages of this application will become apparent and readily understood in conjunction with the following description of the embodiments in conjunction with the accompanying drawings, wherein:

[0051] Figure 1 This is a flowchart illustrating the delayed block embedding processing method for long text provided in the embodiments of this application;

[0052] Figure 2 This is a schematic diagram illustrating a scenario example provided in the embodiments of this application;

[0053] Figure 3 This is a schematic diagram of the structure of the delayed block embedding processing device for long text provided in the embodiments of this application;

[0054] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0056] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0057] The following description, in conjunction with the accompanying drawings, details the delayed block embedding processing method, apparatus, and storage medium for long text provided in this application through specific embodiments and application scenarios.

[0058] The delayed block embedding method for long texts can be applied to terminals, specifically executed by hardware or software within the terminal.

[0059] The terminal includes, but is not limited to, portable communication devices such as mobile phones or tablets with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads). It should also be understood that, in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer with touch-sensitive surfaces (e.g., touchscreen displays and / or touchpads).

[0060] The following embodiments describe a terminal including a display and a touch-sensitive surface. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.

[0061] The delayed segmentation and embedding method for long text provided in this application embodiment can be executed by an electronic device or a functional module or entity in an electronic device that can implement the delayed segmentation and embedding method for long text. The electronic devices mentioned in this application embodiment include, but are not limited to, mobile phones, tablets, computers, cameras, and wearable devices. The delayed segmentation and embedding method for long text provided in this application embodiment will be described below using an electronic device as the execution subject.

[0062] like Figure 1 As shown, the delayed block embedding processing method for this long text includes steps 110, 120 and 130.

[0063] Step 110: Embed the target text to obtain the embedding vector sequence corresponding to the target text.

[0064] In this embodiment of the application, the target text is the original text data that needs to be processed. The target text can be various forms of text content, such as a news report, a novel, webpage content, or even an academic paper. For example, when processing an academic paper, the target text can be the entire content of the paper, including the abstract, introduction, main body, conclusion, etc.

[0065] Embedding is the process of mapping words, sentences, or the entire text into vectors in a high-dimensional space. These vectors capture semantic information in the text, enabling computers to better understand and process text data. Vectors that are close in distance in the vector space correspond to texts that are semantically similar. For example, apples and bananas both belong to the fruit category, and after embedding, their vectors will be relatively close in space.

[0066] Embedding processing can include three main parts: token embedding, position embedding, and segment embedding. Token embedding maps each token to a low-dimensional vector space, position embedding adds information about the token's position in the sequence, and segment embedding is used to distinguish different sentences or text segments.

[0067] In some embodiments, the target text can be embedded using word embedding models such as Word2Vec or GloVe, or it can be embedded using sentence embedding and text embedding. This application does not limit the specific methods used.

[0068] Specifically, the process of embedding target text involves converting words, sentences, symbols, etc., in the text into corresponding embedding vectors, and arranging them according to the order of the text to form an embedding vector sequence. For example, suppose the target text is the sentence "Natural language processing is interesting". First, a pre-trained word embedding model, such as Word2Vec, is needed. Then, each word in the sentence is converted into a fixed-dimensional embedding vector using the Word2Vec model. For example, "natural" is converted into embedding vector v1, "language" into embedding vector v2, "processing" into vector v3, "very" into vector v4, and "interesting" into vector v5. These embedding vectors are then arranged according to their order in the sentence to form an embedding vector sequence [v1, v2, v3, v4, v5].

[0069] In some embodiments, to capture context information, a Transformer-based pre-trained model such as BERT (Bidirectional Encoder Representations from Transformer) can be used. BERT can dynamically generate vector representations of words based on their context in a sentence. For example, for the word "apple" in the sentences "The apple is red" and "The company Apple is big", BERT will generate different vector representations because their contexts are different. When using BERT for embedding processing, first tokenize the target text, and then add special tokens [CLS] and [SEP] to form the input sequence. Input the input sequence into the BERT model, and the BERT model will output the hidden state vectors of each word. These vectors are the embedding vectors that take context information into account. For the entire target text, a corresponding sequence of embedding vectors can be obtained, where each vector contains the semantic information of the word and its context.

[0070] In some embodiments, embedding the target text to obtain a sequence of embedding vectors corresponding to the target text includes:

[0071] Tokenize the target text through an embedding model to obtain a sequence of tokens;

[0072] Perform embedding processing on the sequence of tokens to obtain a sequence of embedding vectors.

[0073] In this embodiment, tokenization is the process of decomposing the target text into smaller units, i.e., tokens. Tokens are the basic units of text processing and can be words, characters, etc. For example, in English, a sentence "Naturallanguage processingis fascinating" can be tokenized into a sequence of tokens "[Natural,language,processing,is,fascinating]". In Chinese, since there are no obvious word delimiters, tokenization is more complex. For example, the sentence "自然语言处理很有趣" can be tokenized into "[自然,语言,处理,很,有趣]". The purpose of tokenization is to convert the text into a form that can be processed by a computer while preserving the semantic information of the text. The embedding model can have a built-in tokenization tool or rely on an external tokenization library to complete the tokenization step. For example, the BERT model can use the WordPiece tokenization algorithm to decompose the text into a sequence of tokens.

[0074] In one example, the transformers library in Python can be used to load any long context embedding model that supports average pooling and longer context inputs, such as jina-embeddings-v3, nomic-v1, etc., to segment and embed the target text.

[0075] In this embodiment, the target text is segmented by an embedding model to obtain a word sequence. The word sequence is then embedded to map discrete words into a continuous vector space, so that the semantic information of the text is mapped into a numerical vector form that can be processed by a computer, thus better preserving the semantic information of the text.

[0076] In some embodiments, embedding processing is performed on the word sequence to obtain an embedding vector sequence, including:

[0077] If the length of the word sequence is greater than the maximum processing length of the embedding model, set a sliding window with a target length;

[0078] A sliding window is slid across the word sequence with a target step size, and each word within the sliding window is embedded during the sliding process; wherein, words in adjacent sliding windows have at least partial overlap.

[0079] In this embodiment, in natural language processing tasks, embedding models typically have a maximum processing length limit because the model's architecture and computational resources determine the length of text it can process at one time. For example, the maximum token size for jina-embeddings-v3, nomic-v1, etc., is around 8k.

[0080] To enable embedding models to better handle word sequences, long texts can be decomposed into multiple processable segments. Specifically, a sliding window with a target length (any length less than the maximum processing length of the embedding model) can be set. By sliding the window across the word sequence, embedding is performed only on the words within the sliding window at a time, thus decomposing the target text into multiple processable segments.

[0081] The target step size is the distance the sliding window moves in each iteration. The choice of step size affects processing efficiency and the accuracy of the results. A smaller step size results in more overlap between sliding windows, allowing for more detailed capture of semantic information in the text, but it also increases computational cost. A larger step size reduces computational cost, but may result in the loss of some important semantic information.

[0082] Specifically, during the sliding window's movement, each word within the window is extracted and input into the embedding model for embedding. The embedding model generates an embedding vector for each word within the sliding window, arranged in the order of the words within the window to form an embedding vector sequence. For example, assuming the word sequence length is 1024, the sliding window length is 512, and the step size is 256, then the word sequence in the first sliding window consists of the first 512 words, and the embedding model generates an embedding vector sequence for these 512 words. Then, the sliding window moves 256 words to the right, and the word sequence in the second sliding window consists of words from the 257th to the 768th word, and the embedding model generates another embedding vector sequence for these 512 words. This process repeats until the entire word sequence has been processed. The embedding vector sequences generated from each sliding window movement are then integrated sequentially to obtain the final embedding vector sequence corresponding to the target text. Adjacent sliding windows refer to sliding windows that move in two consecutive steps.

[0083] It's important to note that the target step size should be smaller than the target length of the sliding window, ensuring at least partial overlap between lexical units in adjacent sliding windows. For example, consider the lexical sequence "[nature, language, processing, very, interesting, of, technology]", with a target window length of 3 and a target step size of 2. The first sliding window contains the lexical sequence "[nature, language, processing]", and the second sliding window contains the lexical sequence "[language, processing, very]". As you can see, the lexical units "[language, processing]" overlap between adjacent windows. This overlap allows the model to utilize the semantic information of the lexical units "language" and "processing" already processed in the first sliding window when processing the second sliding window, thus more accurately understanding the semantics of the lexical unit "very".

[0084] In this embodiment, when the length of the lexical sequence exceeds the maximum processing length of the embedding model, a sliding window of a target length is set and slides on the lexical sequence with a target step size. This allows the long lexical sequence to be divided into multiple processable parts, and the lexical units in adjacent windows at least partially overlap. This reduces the problem of semantic loss caused by the disruption of semantic continuity between adjacent parts of the text during the division process. This enables the model to capture the semantic associations across windows when processing the lexical units in each sliding window, thereby improving the accurate representation of the overall semantics of the long text.

[0085] Step 120: Based on the structure of the target text, determine the block positions and lengths of the multiple text blocks obtained by the block division according to the block division strategy.

[0086] In this embodiment, the chunking strategy is a method of dividing the target text into multiple shorter text blocks. The core objective of the chunking strategy is to preserve the semantic coherence and structural information of the text as much as possible while meeting the processing capabilities of the model. The chunking strategy can be based on the natural structural features of the text, such as sentence boundaries and paragraph boundaries, or according to the semantic units of the text content. For example, a news report can be chunked by paragraphs because each paragraph usually revolves around a theme and has high semantic coherence. For a long sentence, it can be chunked according to clauses or phrases within the sentence.

[0087] In some embodiments, the chunking strategy includes chunking based on the length of the text block, where the length of the text block is the number of tokens contained in each text block. This method divides the target text into text blocks of a fixed token length. For example, the target text can be divided into multiple text blocks with a token length of 256. This method is simple and direct, and can quickly decompose the target text into multiple text blocks.

[0088] Chunking strategies can also include sentence-based chunking, which divides the target text into chunks containing a fixed number of sentences. For example, the target text can be divided into segments where each chunk contains five sentences. This approach better preserves the semantic integrity of sentences because each chunk contains a complete sentence, resulting in high semantic coherence for each chunk.

[0089] Chunking strategies can also include punctuation-based chunking, which utilizes punctuation marks as natural semantic demarcation points, such as periods, question marks, and exclamation marks. For example, the target text can be divided into segments containing one or more sentences, with sentence boundaries determined by periods, question marks, or exclamation marks. Since punctuation marks are usually natural end markers for sentences, this approach can better preserve the semantic integrity of sentences.

[0090] Chunking strategies can also include regular expression-based chunking, which divides text into blocks based on specific patterns or rules, such as specific keywords, phrases, or grammatical structures. For example, regular expressions can be used to match specific patterns in text, such as the "title + content" structure, or to divide text into blocks based on specific keywords. This approach offers high flexibility, allowing chunking rules to be defined according to specific task requirements and text characteristics.

[0091] Chunking strategies can also include paragraph-based chunking, which treats one or more paragraphs as independent text blocks. This is because paragraphs are typically natural structural units within text that revolve around a single theme. For example, a news report usually consists of multiple paragraphs, each with a clear theme and semantic structure. Treating one or more paragraphs as a text block better preserves the semantic integrity of the paragraphs, resulting in higher semantic coherence for each text block. Of course, chunking strategies can take other forms, and this application does not limit these to specific examples.

[0092] In this embodiment, the segmentation position and the length of the multiple text blocks obtained by segmenting the target text can be determined according to the segmentation strategy. Specifically, if the segmentation strategy is based on the number of sentences, the segmentation position is the sentence-ending punctuation mark of a specific number of sentences, and the length of the text block is the number of tokens contained in that specific number of sentences.

[0093] Step 130: Divide the embedding vector sequence based on the block position and the length of multiple text blocks, and perform average pooling on the multiple sub-embedded vector sequences obtained after the division to obtain the embedding vectors corresponding to multiple text blocks.

[0094] In this embodiment of the application, by dividing the embedding vector sequence, multiple sub-embedded vector sequences can be obtained, and each sub-embedded vector sequence corresponds to a text block.

[0095] Specifically, the block position and text block length are the criteria for dividing the embedding vector sequence. For example, suppose the target text is a paragraph containing multiple sentences. After processing with a block strategy, two block positions are determined, dividing the entire paragraph into three text blocks, each with lengths of 256, 300, and 200 tokens respectively. Assuming the embedding model generates a 128-dimensional embedding vector for each token, then the embedding vector sequence for the entire paragraph is a vector sequence of length 756 (256+300+200), with each vector having a dimension of 128. Based on the block position and text block length, this embedding vector sequence can be divided into three sub-embedded vector sequences: the first sub-sequence contains the first 256 embedding vectors, corresponding to the first text block; the second sub-sequence contains the next 300 embedding vectors, corresponding to the second text block; and the third sub-sequence contains the last 200 embedding vectors, corresponding to the third text block.

[0096] In some embodiments, the embedded vector sequence is divided based on the block position and the length of multiple text blocks, including:

[0097] Obtain the embedding vector identifier corresponding to the block position in the embedding vector sequence;

[0098] The embedding vector sequence is divided based on the embedding vector identifier and the length of multiple text blocks.

[0099] The segmentation strategy determines the segmentation positions of the target text. The segmentation positions are determined based on the character positions, sentence or paragraph boundaries, etc. Since the embedding vector sequence is generated based on tokens, it is necessary to map the segmentation positions to the word sequence to determine the corresponding positions in the embedding vector sequence.

[0100] Specifically, during word segmentation, not only is a sequence of lexical units generated, but the start and end positions of each lexical unit in the original text also need to be recorded. For example, suppose the target text is a long sentence: "Natural language processing is a very interesting discipline, involving multiple fields." The commas in the sentence divide the target text into two blocks, with the block positions being the positions of the commas. After word segmentation, the lexical unit sequence is represented as: ["natural", "language", "processing", "is", "one", "door", "very", "interesting", "of", "discipline", ",", "it", "involves", "multiple", "fields", "."]

[0101] The comma "," is the 11th token in the token sequence. Therefore, the token ID corresponding to the block position is 11. The first text block contains tokens with IDs from 0 to 10, and the second text block contains tokens with IDs from 12 to the last token.

[0102] In the embedding model, there is a one-to-one correspondence between the embedding vector sequence and the word sequence. After each token undergoes embedding processing, a corresponding embedding vector representation is generated. Therefore, the tokenid of a token in the word sequence is the embedding vector identifier of that token in the embedding vector sequence.

[0103] After determining the embedding vector identifiers and the length of the text blocks, the embedding vector sequence can be divided. For example, for the embedding vector sequence [v1,v2,v3,v4,v5,v6,v7,v8,v9,v...],... 10 Given that the token IDs corresponding to the block positions are 4 and 7, and the text block lengths are 4, 3, and 3, the partitioning result is as follows: The first text block contains 4 tokens, with the corresponding embedding vector sequence [v1, v2, v3, v4]; the second text block contains 3 tokens, with the corresponding embedding vector sequence [v5, v6, v7]; and the third text block contains 3 tokens, with the corresponding embedding vector sequence [v8, v9, v1, v2, v3, v4]. 10 ].

[0104] In this embodiment, by obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence, the starting position of the block can be accurately located. The embedding vector sequence is divided by combining the length of multiple text blocks, so that each sub-embedded vector sequence corresponds to a text block. This division method not only ensures the consistency between the text block and the embedding vector sequence, but also makes full use of the text semantic information retained by the target text embedding process, so that the divided embedding vector sequence can accurately represent the semantic features of the corresponding text block.

[0105] In some embodiments, obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence includes:

[0106] When the segmentation strategy is based on punctuation or paragraph, the segmentation location is the sentence-ending punctuation mark or paragraph mark.

[0107] Determine the target embedding vector identifier corresponding to the sentence end punctuation mark or paragraph mark in the embedding vector sequence.

[0108] In this embodiment, for punctuation-based segmentation, the segmentation location can be the sentence-ending punctuation mark, such as a period, question mark, exclamation mark, etc. For paragraph-based segmentation, the segmentation location can be the paragraph-ending marker, such as a line break or paragraph separator.

[0109] After determining the block location, the block location can be mapped to the embedding vector sequence. For example, for punctuation-based blocks, the token ID corresponding to the sentence-ending punctuation mark in the word sequence can be determined, thereby determining the corresponding target embedding vector identifier; for paragraph-based blocks, the token ID corresponding to the paragraph marker in the word sequence can be determined, thereby determining the corresponding target embedding vector identifier.

[0110] In this embodiment, when the segmentation strategy is based on punctuation or paragraph, the sentence-ending punctuation mark or paragraph marker is determined as the segmentation position. This fully utilizes the structural characteristics of the text, ensuring that each segment remains relatively independent and complete semantically, and accurately locates the boundaries of the segments.

[0111] In the embodiments of this application, although the segmented sub-embedded vector sequence retains the local structure of the target text, each sub-embedded vector sequence usually contains multiple embedded vectors. These embedded vectors can be aggregated into a fixed-dimensional vector representation by average pooling for subsequent processing.

[0112] Average pooling is a vector aggregation method. The basic idea is to calculate the average of all embedding vectors in a sub-embedded vector sequence to obtain a fixed-length embedding vector. Specifically, for each sub-embedded vector sequence, all embedding vectors in the sequence can be added element by element, and then divided by the number of vectors to obtain an average vector. This average vector can be regarded as the semantic representation of the text block, that is, the embedding vector corresponding to the text block.

[0113] According to the delayed block embedding method for long texts in this application, the target text is embedded to obtain an embedding vector sequence. Based on the structure of the target text, the block positions and lengths of the resulting text blocks are determined according to a block strategy. The embedding vector sequence is then divided based on the block positions and lengths of the text blocks, and average pooling is performed on the resulting sub-embedded vector sequences to obtain embedding vectors corresponding to multiple text blocks. This embodiment of the application first performs overall embedding processing on the target text to obtain an overall embedding vector sequence, and then divides the embedding vector sequence according to a block strategy. This delayed block embedding method can better preserve the semantic information of each text block, reducing the semantic loss problem caused by directly dividing the text into blocks and then embedding in traditional methods. Furthermore, through average pooling, local features within text blocks can be smoothly fused, making the generated embedding vectors more accurately reflect the core semantics of the text blocks, thereby improving the accuracy of the model in long text processing tasks.

[0114] In some embodiments, the method further includes:

[0115] Obtain user problem information;

[0116] Based on the question information, target embedding vectors related to the question information are matched from the semantic vector library; the semantic vector library stores embedding vectors corresponding to multiple text blocks;

[0117] The target embedding vector is converted into text content and combined with question information to construct a prompt message;

[0118] The prompt information is input into the large language model, and the answer information corresponding to the question information output by the large language model is obtained.

[0119] In this embodiment, the embedding vectors corresponding to the obtained text blocks can be stored in a semantic vector library. The semantic vector library is a database that stores embedding vectors corresponding to multiple text blocks obtained from different target texts; these embedding vectors can capture the semantic information of the text blocks.

[0120] Based on the semantic vector library, a question-answering system based on RAG (Retrieval-Augmented Generation) can be built.

[0121] In a question-answering system, users can input question information through text boxes, voice input, or other interactive methods. For example, a user might input a question: "What are the main application areas of natural language processing?" After obtaining the user's question information, it is necessary to understand the semantics of the question and find relevant text content. Specifically, the user's question information can be converted into an embedding vector using an embedding model. The similarity between the question embedding vector and each embedding vector in the semantic vector library is calculated using similarity metrics such as cosine similarity or Euclidean distance. The embedding vector with the highest similarity in the semantic vector library is the target embedding vector, representing the text block most relevant to the question information.

[0122] After finding the target embedding vector, it needs to be converted back into text content. Specifically, this can be done using a decoder or a generative model to map the embedding vector back to the original text. For example, if the target embedding vector represents a text block about a natural language processing application area, the decoder will convert the target embedding vector into specific text content, such as "Natural language processing is applied to multiple fields such as machine translation, sentiment analysis, and speech recognition."

[0123] The converted text content can be combined with the user's question information to construct a prompt, thereby providing sufficient contextual information for the large language model to generate an accurate answer. For example, the prompt could be:

[0124] User's question: What are the main application areas of natural language processing?

[0125] Related text (text content corresponding to the target embedding vector): Natural language processing is applied to multiple fields such as machine translation, sentiment analysis, and speech recognition.

[0126] Such prompts include both the user's question and relevant background information, which helps the large language model better understand the question and generate an answer.

[0127] The constructed prompts are then input into a large language model. For example, inputting the above prompts into a large language model might result in the model outputting the answer: The main application areas of natural language processing include machine translation, sentiment analysis, speech recognition, and text generation. These applications play a vital role in various industries, such as healthcare, finance, and education.

[0128] In one example, such as Figure 2As shown, the target text can be input into the embedding model. Within the embedding model, the target text is segmented into words, generating an embedding vector for each token. Based on a segmentation strategy, the target text is divided into multiple text blocks, such as text block 1, text block 2, ..., text block N. Average pooling is performed on the sub-embedded vector sequence corresponding to each text block to obtain new vector representations, such as the embedding vector of text block 1, the embedding vector of text block 2, and the embedding vector of text block N. The embedding vectors of the text blocks are then stored in a semantic vector library.

[0129] When a user performs a query, the user's question information can be input into the embedding model to generate a corresponding embedding vector. Then, target embedding vectors related to the question information are matched in a semantic vector library. Based on the large language model and prompt information, a RAG retrieval chain is constructed using Langchain, ultimately outputting the answer information corresponding to the question.

[0130] In this embodiment, by storing the embedding vectors corresponding to multiple text blocks in the semantic vector library, and matching the target embedding vectors related to the question information from the semantic vector library, the content most relevant to the user's question can be quickly located. The target embedding vectors are converted into text content, which is used as information reference. Combined with the question information, prompt information is constructed. By utilizing the powerful language generation capabilities of the large language model, the answer information corresponding to the question information is obtained, thus achieving an accurate answer to the user's question.

[0131] The delayed segmentation embedding method for long text provided in this application can be executed by a delayed segmentation embedding device for long text. This application uses the delayed segmentation embedding device for long text executing the delayed segmentation embedding method for long text as an example to illustrate the delayed segmentation embedding device for long text provided in this application.

[0132] This application also provides a delayed block embedding processing device for long text.

[0133] like Figure 3 As shown, the delayed chunking embedding processing device for the long text includes:

[0134] The embedding module 310 is used to embed the target text to obtain the embedding vector sequence corresponding to the target text;

[0135] The determination module 320 is used to determine the block positions and lengths of the multiple text blocks obtained by the block division based on the structure of the target text according to the block division strategy.

[0136] The partitioning module 330 is used to partition the embedding vector sequence based on the block position and the length of multiple text blocks, and to perform average pooling on the multiple sub-embedded vector sequences obtained after partitioning to obtain the embedding vectors corresponding to multiple text blocks.

[0137] According to the delayed block embedding processing apparatus for long text of this application, the target text is embedded to obtain an embedding vector sequence corresponding to the target text; the block positions and lengths of the multiple text blocks obtained by the block division are determined based on the structure of the target text according to the block strategy; the embedding vector sequence is divided based on the block positions and the lengths of the multiple text blocks, and the multiple sub-embedded vector sequences obtained after the division are averaged to obtain the embedding vectors corresponding to multiple text blocks. This embodiment of the application first performs overall embedding processing on the target text to obtain an overall embedding vector sequence, and then divides the embedding vector sequence according to the block strategy. This delayed block embedding processing method can better preserve the semantic information of each text block, reducing the semantic loss problem caused by directly dividing the text into blocks and then embedding in traditional methods. Furthermore, through the average pooling operation, local features within the text blocks can be smoothly fused, making the generated embedding vectors more accurately reflect the core semantics of the text blocks, thereby improving the accuracy of the model in long text processing tasks.

[0138] In some embodiments, the delayed chunking embedding processing apparatus for long text further includes:

[0139] The question-answering module is used to obtain the user's question information; match the target embedding vectors related to the question information from the semantic vector library; the semantic vector library stores the embedding vectors corresponding to multiple text blocks; convert the target embedding vectors into text content and combine them with the question information to construct prompt information; input the prompt information into the large language model to obtain the answer information corresponding to the question information output by the large language model.

[0140] In some embodiments, the embedding module 310 is further configured to:

[0141] The target text is segmented using an embedding model to obtain a word sequence;

[0142] The word sequence is embedded to obtain an embedding vector sequence.

[0143] In some embodiments, the embedding module 310 is further configured to:

[0144] If the length of the word sequence is greater than the maximum processing length of the embedding model, set a sliding window with a target length;

[0145] A sliding window is slid across the word sequence with a target step size, and each word within the sliding window is embedded during the sliding process; wherein, words in adjacent sliding windows have at least partial overlap.

[0146] In some embodiments, the partitioning module 330 is further configured to:

[0147] Obtain the embedding vector identifier corresponding to the block position in the embedding vector sequence;

[0148] The embedding vector sequence is divided based on the embedding vector identifier and the length of multiple text blocks.

[0149] In some embodiments, the partitioning module 330 is further configured to:

[0150] When the segmentation strategy is based on punctuation or paragraph, the segmentation location is the sentence-ending punctuation mark or paragraph mark.

[0151] Determine the target embedding vector identifier corresponding to the sentence end punctuation mark or paragraph mark in the embedding vector sequence.

[0152] The delayed segmentation and embedding processing device for long text in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the specific implementation.

[0153] The delayed segmentation and embedding processing device for long text in this application embodiment can be a device with an operating system. This operating system can be a Microsoft (Windows) operating system, an Android operating system, an iOS operating system, or other possible operating systems; this application embodiment does not specifically limit it.

[0154] In some embodiments, such as Figure 4As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements the various processes of the above-described long text delayed block embedding processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0155] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0156] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described long text delayed block embedding processing method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0157] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described delayed block embedding processing method for long text.

[0159] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0160] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described long text delayed block embedding processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0161] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0162] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0163] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0164] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0165] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0166] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for delayed block embedding of long text, characterized in that, include: The target text is embedded to obtain the embedding vector sequence corresponding to the target text; Based on the structure of the target text, the block positions and lengths of the resulting text blocks are determined according to the block segmentation strategy. The embedding vector sequence is divided based on the block position and the length of the multiple text blocks, and the multiple sub-embedded vector sequences obtained after the division are averaged to obtain the embedding vectors corresponding to the multiple text blocks.

2. The method according to claim 1, characterized in that, The method further includes: Obtain user problem information; Based on the question information, target embedding vectors related to the question information are matched from a semantic vector library; the semantic vector library stores the embedding vectors corresponding to the plurality of text blocks; The target embedding vector is converted into text content and combined with the question information to construct a prompt message; The prompt information is input into the large language model to obtain the answer information corresponding to the question information output by the large language model.

3. The method according to claim 1, characterized in that, The embedding process of the target text to obtain the embedding vector sequence corresponding to the target text includes: The target text is segmented using an embedding model to obtain a word sequence; The word sequence is embedded to obtain the embedding vector sequence.

4. The method according to claim 3, characterized in that, The embedding process of the word sequence to obtain the embedding vector sequence includes: If the length of the lexical sequence is greater than the maximum processing length of the embedding model, a sliding window of target length is set; The sliding window on the word sequence is slid with a target step size, and the word elements in the sliding window during the sliding process are embedded respectively; wherein the word elements in adjacent sliding windows have at least partial overlap.

5. The method according to claim 1, characterized in that, The segmentation strategy includes segmentation based on the length of the text block, segmentation based on the number of sentences, segmentation based on punctuation marks, segmentation based on regular expressions, or segmentation based on paragraphs.

6. The method according to claim 5, characterized in that, The step of dividing the embedded vector sequence based on the block position and the length of the multiple text blocks includes: Obtain the embedding vector identifier corresponding to the block position in the embedding vector sequence; The embedding vector sequence is divided according to the embedding vector identifier and the length of the plurality of text blocks.

7. The method according to claim 6, characterized in that, The step of obtaining the embedding vector identifier corresponding to the block position in the embedding vector sequence includes: When the segmentation strategy is based on punctuation or paragraph, the segmentation position is the sentence end punctuation mark or paragraph mark; Determine the target embedding vector identifier corresponding to the sentence end punctuation mark or the paragraph mark in the embedding vector sequence.

8. A delayed chunking and embedding processing apparatus for long text, characterized in that, include: The embedding module is used to embed the target text to obtain the embedding vector sequence corresponding to the target text; The determination module is used to determine the block positions and lengths of the multiple text blocks obtained by the block division based on the structure of the target text according to the block division strategy. The partitioning module is used to partition the embedding vector sequence based on the block position and the length of the multiple text blocks, and to perform average pooling on the multiple sub-embedded vector sequences obtained after partitioning to obtain the embedding vectors corresponding to the multiple text blocks.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the delayed block embedding processing method for long text as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the delayed block embedding processing method for long text as described in any one of claims 1-7.

Images