Long text generation alignment method and system based on synthetic position coding

Abstract

The invention provides a synthesis position coding-based long text generation alignment method and system, and relates to the technical field of natural language processing and text processing, and the method comprises the following steps: obtaining dialogue type document data, and splitting the dialogue type document data into continuous instruction-response pair sequences; encoding position information of lexical elements in the instruction-response pair sequence through a position index; expanding the relative distance of the semantic dependence of the instruction-response on the sequence, redistributing a position index to expand the original position, determining whether the condition of adding the skipping step is met by using an indicator function, and determining when to set the step of introducing the skipping; a skipping step is introduced, a skipping distance is selectively applied to a specific structure in the position index, and the contextual semantic dependency relationship of the instruction-response pair sequence is expanded; and inputting the expanded position index into an alignment model based on LLaMA-2 to realize alignment synthesis of the long text.

Description

Technical Field

[0001] The present disclosure relates to the technical field of natural language processing and text processing, and in particular to a method and system for generating and aligning long texts based on synthetic position coding. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence technology, the ability of large language models (LLMs) to process and understand long texts has become increasingly important. These models play a role in a variety of complex practical applications, such as answering questions or summarizing, understanding, and processing code bases from multiple documents. In order to meet these application requirements, the context window of large speech models is constantly expanding. For example, through continuous pre-training, a context window of 128K tokens can be achieved.

[0004] Despite the progress in expanding the context window, how to enable LLMs to leverage their long text capabilities to interpret long and complex instructions remains an underexplored area. A major obstacle is the lack of high-quality, open-source long instruction datasets and the challenges associated with annotating such data. A promising approach is to synthesize common short instructions into long instruction samples.

[0005] However, existing methods mainly focus on simply extending the length of a given sample, ignoring the more critical aspect of effectively building long-distance dependencies. Methods such as LongChat and LongLLAMA create longer samples by connecting shorter samples. The long-distance dependencies built by these strategies come from unrelated samples and may not be able to effectively simulate the long-distance dependencies required for long text tasks. Summary of the invention

[0006] In order to solve the above problems, the present disclosure proposes a long text generation and alignment method and system based on synthetic position encoding, which uses the position index of short instructions to create samples of long-distance dependencies, and inserts skip steps in the position index of instruction-response pairs, thereby enhancing the model's ability to process long texts without generating a large amount of data or modifying the architecture.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] Long text generation and alignment method based on synthetic position encoding, including:

[0009] Acquire conversational document data, and split the conversational document data into a continuous sequence of command-response pairs;

[0010] The position information of the word unit in the command-response sequence is encoded by the position index;

[0011] Expanding the relative distance of the semantic dependency of the instruction-response sequence, reallocating the position index to expand the original position, determining whether the condition of adding a skip step is met by using an indicator function, and determining when to set the introduced skip step;

[0012] Introducing a skip step to selectively apply the skip distance to specific structures in the position index, expanding the contextual semantic dependencies of the instruction-response pair sequence;

[0013] The expanded position index is input into the LLaMA-2-based alignment model to achieve alignment synthesis of long texts.

[0014] According to some embodiments, the present disclosure adopts the following technical solutions:

[0015] A long text generation and alignment system based on synthetic positional encoding, including:

[0016] A data acquisition module, used for acquiring conversational document data and splitting the conversational document data into a continuous sequence of command-response pairs;

[0017] A position encoding module, for encoding position information of a word unit in a command-response pair sequence by a position index;

[0018] An extended skip module is used to expand the relative distance of the semantic dependency of the instruction-response sequence, reallocate the position index to expand the original position, determine whether the condition of adding a skip step is met by using an indicator function, and determine when to set the introduced skip step; introduce the skip step, selectively apply the skip distance to a specific structure in the position index, and expand the contextual semantic dependency of the instruction-response sequence;

[0019] The alignment module is used to input the expanded position index into the LLaMA-2-based alignment model to achieve alignment synthesis of long texts.

[0020] According to some embodiments, the present disclosure adopts the following technical solutions:

[0021] A computer program product includes a computer program, and when the computer program is executed by a processor, the long text generation and alignment method based on synthetic position coding is implemented.

[0022] According to some embodiments, the present disclosure adopts the following technical solutions:

[0023] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the long text generation and alignment method based on synthetic position coding is implemented.

[0024] According to some embodiments, the present disclosure adopts the following technical solutions:

[0025] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the long text generation and alignment method based on synthetic position coding.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] The present invention discloses a long text generation alignment method based on synthetic position encoding, and proposes the SkipAlign method. The proposal of SkipAlign is based on a core assumption that expanding the relative distance of such semantic structures to include longer scales is crucial to unlocking the long text capabilities of the language model. SkipAlign achieves this by strategically modifying position indexes. By selectively skipping certain position indexes in the instruction-following sample, the present invention is able to expand the relative distance of semantic dependencies and create long-distance dependencies.

[0028] The long text generation alignment method based on synthetic position encoding disclosed in the present invention, SkipAlign, improves the long text capability of the model by simulating long-distance dependencies; SkipAlign avoids the additional long data required to train or modify the large language model architecture, making it a computationally efficient solution; SkipAlign also helps large language models process inputs beyond their original context window. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0030] Figure 1 It is a schematic diagram of the SkipAlign method of an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0033] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0034] Example 1

[0035] In one embodiment of the present disclosure, a long text generation alignment method based on synthetic position coding is provided, comprising:

[0036] Step 1: Obtain conversational document data, and split the conversational document data into a continuous sequence of command-response pairs;

[0037] Step 2: Encode the position information of the word unit in the command-response sequence through the position index;

[0038] Step 3: Expand the relative distance of the semantic dependency of the instruction-response sequence, reallocate the position index to expand the original position, use an indicator function to determine whether the conditions for adding a skip step are met, and determine when to set the introduced skip step;

[0039] Step 4: Introduce a skip step to selectively apply the skip distance to specific structures in the position index to expand the contextual semantic dependencies of the instruction-response sequence;

[0040] Step 5: Add the expanded position index to the conversation data as additional input to achieve aligned synthesis of long texts.

[0041] As an embodiment, the long text generation alignment method based on synthetic position encoding disclosed in the present invention proposes a new method called SkipAlign, which uses the position index of short instructions to create meaningful samples of long-distance dependencies. SkipAlign strategically inserts some skip steps in the position index of the instruction-response pair, thereby enhancing the model's ability to process long texts without generating a large amount of data or modifying the architecture. The core idea of ​​SkipAlign is that it is the key to unlocking the long text capability of the language model by extending the relative distance of the semantic structure to a longer scale. SkipAlign creates long-distance dependencies by selectively skipping certain position indexes and extending the relative distance of semantic dependencies in the instruction-following sample. The present invention uses the Tülü V2 dataset, which is a high-quality dataset mixed with manually annotated and GPT-generated dialogue data. This dataset provides rich and diverse resources for model training. According to their settings, the input sample is truncated to 4096 tokens. For SkipAlign, additional position indexes are introduced during preprocessing. The parameters of SkipAlign are as follows: the maximum extension length is set to 100K, the subsampling ratio is 0.5, and the default skip strategy is Skip-Outer. The specific implementation process is as follows:

[0042] Step 1: Obtain conversational document data, and split the conversational document data into a continuous sequence of command-response pairs;

[0043] Conversational document data is obtained as samples, and these samples are organized into a continuous sequence of command-response pairs. Specifically, data from existing open source conversational document datasets such as Ultrachat, Tulu, and Sharegpt can be obtained as samples and organized into a continuous sequence of command-response pairs.

[0044] Formally speaking, let m = (x1, y1, ..., x i ,y i ) represents a sequence of i rounds of command-response pairs. The autoregressive language model is trained using the following objective function:

[0045]

[0046] In this example of a dialogue format, the model’s task is to follow the instruction x j and the sequence of previous pairs, predict the response y i .

[0047] Step 2: Encode the position information of the word unit in the command-response sequence through the position index;

[0048] Specifically, Transformer-based language models supplement input tokens with position information, which is encoded by position indexes. Although a variety of position embedding techniques have been proposed, they generally rely on position indexes to accurately convey the position information of tokens. By default, position indexes are assigned sequentially as (0, 1, ..., |m|-1), where |m| represents the length of the input sequence.

[0049] Step 3: Expand the relative distance of the semantic dependency of the instruction-response sequence, reallocate the position index to expand the original position, use an indicator function to determine whether the conditions for adding a skip step are met, and determine when to set the introduced skip step;

[0050] Specifically, SkipAlign is proposed based on a core assumption that expanding the relative distance of such semantic structures to include longer scales is crucial to unlocking the long text capabilities of language models. SkipAlign achieves this by strategically modifying position indexes. By selectively skipping certain position indexes in instruction-following samples, the present invention is able to expand the relative distance of semantic dependencies and create long-distance dependencies.

[0051] Further, the present invention discloses a method for skipping positions by shifting, and the process is as follows: The goal of the present invention is to extend the relative distance of semantic dependencies in the instruction data set, beyond its maximum sample length l to reach the extended maximum length L. Extending the original position to the interval [0, L] is achieved by reallocating the position index. The present invention regards the instruction or response as a basic unit and moves all their position indexes at the same time.

[0052] Formally, given an i-th round of samples m, let P(m) = (c1, c2, ..., c 2i-1 , c 2i ) represents its original position index, which is the concatenation of the position index of each block in the command-response pair. In P(m), odd and even subscripts correspond to commands and responses, respectively. By shifting each position block to the right, the deviation vector u = (u1, u2, ... u 2i ) to create larger relative positions, where each constant u∈u is a constant bias of the move. By moving different blocks with different biases, skipped positions are created between them. Since the fundamental requirement for valid position indexing is incrementality, this requires minimal shift bias is set to the cumulative shift deviation of the previous token

[0053] Introduce a skip step, denoted as s i , so that s i A value of zero means that c iand its previous c i-1 No skipping occurs between i For regularity, s is introduced between these two positions. i To achieve uniform distribution of relative distances in [0, L] after moving, the present invention samples s from the uniform distribution. i :

[0054]

[0055] in, Indicates the maximum allowed skip length, taking into account the sample length |m| and the position that has been skipped The remaining key task is to design a skip strategy to determine when to set s i >0 to introduce a skip step.

[0056] Step 4: Introduce a skip step to selectively apply the skip distance to specific structures in the position index to expand the contextual semantic dependencies of the instruction-response pair sequence;

[0057] This paper provides three different skip strategies to study the contribution of various semantic dependencies to the model's long text capability. These strategies selectively apply the skip distance to specific structures in the sample to determine when to set s i >0 to introduce a skip step.

[0058] 1) Skip-All: This strategy applies skipping to all roles in the sample without any selection

[0059] 2) Skip-Inner: This strategy only adds skip steps inside the pair, that is, between the instruction and its response. In short, this strategy only adds skip steps inside the pair. i Add s when coming from response i .

[0060] 3) Skip-Outer: This strategy only introduces skipping steps between different dialogue turns. In short, this strategy only introduces skipping steps between different dialogue turns. i Add s when coming from a directive i .

[0061] The present disclosure uses an indicator function DO_SKIP to determine c i Whether the conditions for adding a skip step are met. The function returns 1 if the conditions are met, otherwise it returns 0. In addition, in order to control the number of synthesis positions, the present disclosure extracts p% of valid positions to add skip steps. The overall rules are summarized as follows:

[0062]

[0063] Among them, ∈i is uniformly sampled from [0,1].

[0064] Simulation comparison

[0065] In response to recent progress in extending context windows, this paper investigates the impact of these models on long-context alignment. SFT is performed using two base models with different context window sizes:

[0066] 1. The LLaMA-2 model, whose context window is 4094 tokens, is used as the comparison baseline for this disclosure.

[0067] 2. Yi-6B-200K model, which significantly expands the context window of the Yi-6B model to an impressive 200K tokens through continuous pre-training. For the LLaMA-2-based model, the present disclosure uses the Neural Tangent Kernel (NTK) before training to expand the position embedding to the maximum training or inference length. In contrast, for the Yi-6B-200K model, additional position expansion is unnecessary because the maximum embedding length inherent to the model is already 200K.

[0068] Example 2

[0069] In one embodiment of the present disclosure, a long text generation and alignment system based on synthetic position coding is provided, comprising:

[0070] A data acquisition module, used for acquiring conversational document data and splitting the conversational document data into a continuous sequence of command-response pairs;

[0071] A position encoding module, for encoding position information of a word unit in a command-response pair sequence by a position index;

[0072] An extended skip module is used to expand the relative distance of the semantic dependency of the instruction-response sequence, reallocate the position index to expand the original position, determine whether the condition of adding a skip step is met by using an indicator function, and determine when to set the introduced skip step; introduce the skip step, selectively apply the skip distance to a specific structure in the position index, and expand the contextual semantic dependency of the instruction-response sequence;

[0073] The alignment module is used to input the expanded position index into the LLaMA-2-based alignment model to achieve alignment synthesis of long texts.

[0074] Example 3

[0075] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the long text generation and alignment method based on synthetic position coding is implemented.

[0076] Example 4

[0077] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the long text generation and alignment method based on synthetic position coding is implemented.

[0078] Example 5

[0079] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the long text generation and alignment method based on synthetic position coding.

[0080] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0081] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0082] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A long text generation and alignment method based on synthetic position encoding, characterized in that: include: Acquire conversational document data, and split the conversational document data into a continuous sequence of command-response pairs; The position information of the word unit in the command-response sequence is encoded by the position index; Expanding the relative distance of the semantic dependency of the instruction-response sequence, reallocating the position index to expand the original position, determining whether the condition of adding a skip step is met by using an indicator function, and determining when to set the introduced skip step; Introducing a skip step to selectively apply the skip distance to specific structures in the position index, expanding the contextual semantic dependencies of the instruction-response pair sequence; The expanded position index is input into the LLaMA-2-based alignment model to achieve alignment synthesis of long texts.

2. The long text generation and alignment method based on synthetic position coding as claimed in claim 1, characterized in that: Split the conversational document data into a continuous sequence of command-response pairs, including: Let m = (x1, y1, ..., x i ,y i ) represents a sequence of i rounds of instruction-response pairs, and the autoregressive language model is trained using the following objective function: In this sample dialog format, according to instruction x j and the sequence of previous pairs, predict the response y i .

3. The long text generation and alignment method based on synthetic position coding as claimed in claim 1, characterized in that: The Transformer-based language model uses position information to supplement the input word unit, encodes it through position index, and relies on the position index to represent the position information of the word unit. The position index is assigned in sequence as (0, 1, ..., |m|-1), where |m| represents the length of the input sequence.

4. The long text generation and alignment method based on synthetic position coding as claimed in claim 1, characterized in that: The relative distance of semantic dependency of instruction-response sequence is expanded, and the position index is redistributed to expand the original position, treating the instruction or response as a basic unit and moving all their position indexes at the same time.

5. The long text generation and alignment method based on synthetic position coding as claimed in claim 1, characterized in that: Given an i-round sample m, let P(m) = (c1, c2, ..., c 2i-1 , c 2i ) represents its original position index, which is formed by concatenating the position index of each block in the command-response pair. In P(m), odd and even subscripts correspond to commands and responses, respectively. By shifting each position block to the right, the deviation vector u = (u1, u2, ... u 2i ) to create larger relative positions, where each constant u∈u is a constant deviation of the move, by moving different blocks with different deviations.

6. The long text generation and alignment method based on synthetic position coding as claimed in claim 5, characterized in that: The basic requirement for valid position indexing is incrementality, which requires minimal shift bias. is set to the cumulative shift deviation of the previous token Introduce a skip step, denoted as s i , so that s i A value of zero means that c i and its previous c i-1 No skipping occurs between i To make it regular, we introduce s between these two positions. i The position index is skipped to achieve a uniform distribution of relative distances in [0, L] after the move, and s is sampled from the uniform distribution. i : in, Indicates the maximum allowed skip length, taking into account the sample length |m| and the position that has been skipped Design a skip strategy to determine when to set s i >0 to introduce a skip step.

7. A long text generation and alignment system based on synthetic position encoding, characterized in that: include: A data acquisition module, used for acquiring conversational document data and splitting the conversational document data into a continuous sequence of command-response pairs; A position encoding module, for encoding position information of a word unit in a command-response pair sequence by a position index; An extended skip module is used to expand the relative distance of the semantic dependency of the instruction-response sequence, reallocate the position index to expand the original position, determine whether the condition of adding a skip step is met by using an indicator function, and determine when to set the introduced skip step; introduce the skip step, selectively apply the skip distance to a specific structure in the position index, and expand the contextual semantic dependency of the instruction-response sequence; The alignment module is used to input the expanded position index into the LLaMA-2-based alignment model to achieve alignment synthesis of long texts.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the long text generation and alignment method based on synthetic position coding described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by the processor, the long text generation and alignment method based on synthetic position coding as described in any one of claims 1-6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the long text generation and alignment method based on synthetic position coding as described in any one of claims 1-6.

Images