Language model training method based on context position coding and Fourier transform

Abstract

The invention discloses a language model training method based on context position coding and Fourier transform, which belongs to the technical field of computer natural language processing, and comprises the following steps: S1, preprocessing text data input by a user; s2, performing context position coding on the preprocessed text information; s3, after context position coding, Fourier transform is carried out on each relative position coding vector; s4, performing model pre-training and instruction fine tuning on the position coding vector after Fourier transform, and measuring the difference between a prediction result and a real target by using a minimization loss function; and S5, finally completing model tensor distributed parallel training. According to the method, the accuracy in the model training process is improved, the method is suitable for a long context scene, and accurate text generation can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of computer natural language processing, and in particular to a language model training method based on context position coding and Fourier transform. Background Art

[0002] Currently, the defects of language model training in existing technologies mainly include: 1. The context length supported by model training is relatively short, and accurate text generation in long context scenarios cannot be achieved.

[0003] 2. The model training speed is slow and occupies a high amount of video memory resources.

[0004] 3. The accuracy is low during model training.

[0005] A Chinese patent document with publication number CN116166771 A and publication date May 26, 2023, discloses a method for identifying speech service content based on behavioral text processing, which is characterized by comprising the following steps: S1: Set up a voice collection device to capture the working audio of customer service personnel. The audio between each call and hang-up by the customer service personnel is regarded as a complete service audio, and the audio is divided into frames. The framed audio is clustered based on the voiceprint features of each frame to obtain customer service audio information. S2: Build a customer service personnel voice library and use a voiceprint recognition model to determine the identity of the customer service personnel in the service audio; use a speech recognition model to convert the service audio into speech text; S3: Establish a text keyword library to identify service behaviors; establish a keyword acquisition module to calculate and retrieve behavior keywords in voice text; S4: Establish a speaker behavior recognition model; establish a behavior tag library, train the constructed speaker behavior recognition model, and determine the speaker behavior through the behavior keyword sequence; S5: Establish a service content recognition model; establish a service content tag library, and train the constructed service content recognition model to determine customer service content.

[0006] The patent document discloses a method for identifying speech service content based on action text processing. This method replaces service audio segmentation with action keywords to construct action keyword sequences, improving the accuracy of service action recognition. However, it still cannot achieve accurate text generation for long context scenarios. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art, the present invention provides a language model training method based on context position coding and Fourier transform. The present invention improves the accuracy of the model training process, is applicable to long context scenarios, and can achieve accurate text generation.

[0008] The present invention is achieved through the following technical solutions: The language model training method based on context position coding and Fourier transform is characterized by comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0009] In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0010] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0011] In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

[0012] In step S4, model pre-training refers to enabling the model to learn deep grammatical structures from the mask prediction task through adaptive mask ratio and position control, combined with joint mapping of frequency domain position encoding and contextual semantics.

[0013] In step S4, instruction fine-tuning refers to using labeled data for training and generating responses based on input.

[0014] In step S4, the loss function is a cross entropy loss function.

[0015] In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing.

[0016] In step S4, during the model pre-training and instruction fine-tuning process, the self-attention mechanism is used to process sequence data.

[0017] In step S5, the tensor distributed parallel training refers to dividing the matrix parameters of the model by rows and columns respectively, and distributing them to different graphics processors for matrix operations.

[0018] The beneficial effects of the present invention are mainly manifested in the following aspects: 1. The present invention, S1, pre-processes the text data input by the user; S2, performs context position encoding on the pre-processed text information; S3, after the context position encoding, performs Fourier transform on each relative position encoding vector; S4, performs model pre-training and instruction fine-tuning on the position encoding vector after Fourier transform, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5, finally completes the distributed parallel training of the model tensor, which improves the accuracy of the model training process compared with the existing technology, is suitable for long context scenarios, and can achieve accurate text generation.

[0019] 2. The present invention, by combining text preprocessing, Fourier transform-enhanced position encoding, model pretraining, fine-tuning, and distributed parallel training, achieves efficient representation of contextual position relationships, large-scale model parameter optimization, and collaborative computing across graphics processor resources, ultimately significantly accelerating the training process while improving semantic understanding accuracy.

[0020] 3. In the present invention, in step S3, performing Fourier transform on each relative position coding vector means converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information, which can realize the frequency domain reconstruction of the natural language representation input, generate vector matrix information with orthogonal characteristics, and significantly improve the model training efficiency and prediction accuracy.

[0021] 4. In the present invention, in step S4, model pre-training and instruction fine-tuning are completed through parameter sharing. Parameter sharing can reduce the complexity of the model, improve training efficiency, and avoid overfitting problems.

[0022] 5. In the present invention, in step S4, during the model pre-training and instruction fine-tuning process, a self-attention mechanism is used when processing sequence data. By using the self-attention mechanism, the model is allowed to focus on other words related to the current word when processing sequence data, thereby effectively capturing long-distance dependencies in the text.

[0023] 6. In the present invention, in step S5, tensor distributed parallel training refers to dividing the matrix parameters of the model by rows and columns respectively, and distributing them to different graphics processors for matrix operations, which improves the speed of model training, significantly reduces the memory usage of a single card, effectively improves the computational parallelism, and shortens the overall training time. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments: Figure 1 It is a flowchart of the present invention. DETAILED DESCRIPTION

[0025] Example 1 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0026] This embodiment is the most basic implementation method, S1. Preprocessing the text data input by the user; S2. Context position encoding of the preprocessed text information; S3. After context position encoding, Fourier transform of each relative position encoding vector; S4. Model pre-training and instruction fine-tuning of the position encoding vector after Fourier transformation, using the minimized loss function to measure the difference between the predicted result and the true target; S5. Finally, completing the distributed parallel training of the model tensor. Compared with the existing technology, this improves the accuracy of the model training process, is suitable for long context scenarios, and can achieve accurate text generation.

[0027] Example 2 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0028] Preferably, in step S1, preprocessing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0029] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0030] This embodiment is a preferred implementation method. By combining text preprocessing, Fourier transform-enhanced position encoding, model pre-training, fine-tuning and distributed parallel training, it achieves efficient representation of contextual position relationships, large-scale model parameter optimization and collaborative computing across graphics processor resources, ultimately significantly accelerating the training process while improving the accuracy of semantic understanding.

[0031] Example 3 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0032] In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0033] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0034] In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

[0035] This embodiment is a preferred implementation method. In step S3, performing Fourier transform on each relative position encoding vector means converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information, which can realize the frequency domain reconstruction of the natural language representation input and generate vector matrix information with orthogonal characteristics, significantly improving the model training efficiency and prediction accuracy.

[0036] Example 4 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0037] In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0038] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0039] In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

[0040] Further preferably, in step S4, model pre-training refers to enabling the model to learn deep grammatical structure from the mask prediction task by adaptive mask ratio and position control, combined with joint mapping of frequency domain position encoding and contextual semantics.

[0041] In step S4, instruction fine-tuning refers to using labeled data for training and generating responses based on input.

[0042] In step S4, the loss function is a cross entropy loss function.

[0043] In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing.

[0044] This embodiment is another preferred implementation method. In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing. Parameter sharing can reduce the complexity of the model, improve training efficiency, and avoid overfitting problems.

[0045] Example 5 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0046] In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0047] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0048] In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

[0049] In step S4, model pre-training refers to enabling the model to learn deep grammatical structures from the mask prediction task through adaptive mask ratio and position control, combined with joint mapping of frequency domain position encoding and contextual semantics.

[0050] In step S4, instruction fine-tuning refers to using labeled data for training and generating responses based on input.

[0051] In step S4, the loss function is a cross entropy loss function.

[0052] In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing.

[0053] In step S4, during the model pre-training and instruction fine-tuning process, the self-attention mechanism is used to process sequence data.

[0054] This embodiment is another preferred implementation method. In step S4, during the model pre-training and instruction fine-tuning process, a self-attention mechanism is used when processing sequence data. By using the self-attention mechanism, the model is allowed to pay attention to other words related to the current word when processing sequence data, thereby effectively capturing long-distance dependencies in the text.

[0055] Example 6 See also Figure 1 , a language model training method based on context position coding and Fourier transform, comprising the following steps: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

[0056] In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

[0057] In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

[0058] In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

[0059] In step S4, model pre-training refers to enabling the model to learn deep grammatical structures from the mask prediction task through adaptive mask ratio and position control, combined with joint mapping of frequency domain position encoding and contextual semantics.

[0060] In step S4, instruction fine-tuning refers to using labeled data for training and generating responses based on input.

[0061] In step S4, the loss function is a cross entropy loss function.

[0062] In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing.

[0063] More preferably, in step S4, during the model pre-training and instruction fine-tuning process, a self-attention mechanism is used when processing sequence data.

[0064] In step S5, the tensor distributed parallel training refers to dividing the matrix parameters of the model by rows and columns respectively, and distributing them to different graphics processors for matrix operations.

[0065] This embodiment is the best implementation method. In step S5, tensor distributed parallel training refers to splitting the model's matrix parameters by rows and columns, and distributing them to different graphics processors for matrix operations. This improves the speed of model training, significantly reduces the memory usage of a single card, effectively improves the computational parallelism, and shortens the overall training time.

[0066] The basic principles of the present invention are as follows: First, the user input text is cleaned and standardized, and dynamic context position encoding is injected to capture the positional correlation between words; then, the position vector is mapped from the time domain to the frequency domain through Fourier transform, and multi-scale spectral features are decomposed to enhance the model's perception of long-range dependencies and local position patterns; on this basis, general semantic representations are learned through pre-training and combined with instruction fine-tuning to achieve task adaptation, and the loss function is used to backpropagate to optimize the model parameters; finally, with the help of tensor distributed parallel training, the large model parameter matrix is ​​split into rows and columns to multiple graphics processor clusters, and through computing load balancing and gradient synchronization strategies, efficient training of massive parameters and maximum utilization of hardware resources are achieved, thereby significantly accelerating the training process while improving the accuracy of semantic modeling.

Claims

1. A language model training method based on context position coding and Fourier transform, characterized in that: The following steps are involved: S1. Preprocessing text data input by the user; S2, performing context position encoding on the preprocessed text information; S3. After context position encoding, each relative position encoding vector is subjected to Fourier transform; S4, the position encoding vector after Fourier transformation, performs model pre-training and instruction fine-tuning, and uses the minimization loss function to measure the difference between the predicted result and the true target; S5. Finally, complete the distributed parallel training of the model tensor.

2. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S1, pre-processing the text data input by the user refers to removing interference and noise in the data by filtering and screening.

3. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S2, context position encoding refers to establishing a dynamic screening mechanism using the query features of the current semantic unit and the key features of the historical unit, generating a continuous position offset associated with the context, and using differentiable interpolation operations to perform mixed embedding representation on adjacent discrete position embedding vectors to form a position relationship modeling mechanism based on context semantic perception.

4. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S3, performing Fourier transform on each relative position coding vector refers to converting the time domain convolution operation into a frequency domain product operation to generate vector matrix information.

5. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S4, model pre-training refers to enabling the model to learn deep grammatical structures from the mask prediction task through adaptive mask ratio and position control, combined with joint mapping of frequency domain position encoding and contextual semantics.

6. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S4, instruction fine-tuning refers to using labeled data for training and generating responses based on input.

7. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S4, the loss function is a cross entropy loss function.

8. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S4, model pre-training and instruction fine-tuning are completed through parameter sharing.

9. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S4, during the model pre-training and instruction fine-tuning process, the self-attention mechanism is used to process sequence data.

10. The language model training method based on context position coding and Fourier transform according to claim 1, characterized in that: In step S5, the tensor distributed parallel training refers to dividing the matrix parameters of the model by rows and columns respectively, and distributing them to different graphics processors for matrix operations.

Images