Text processing method, device, equipment, storage and program of long-context large model based on dynamic temperature factor improvement

Abstract

The embodiment of the invention discloses a text processing method, device, equipment, storage and program for a long context large model based on dynamic temperature factor improvement, and the method comprises the steps: obtaining a target data set; wherein the target data set comprises extended length text data; inputting the target data set into a target large model, and performing text processing on the target data set through the target large model to obtain a model processing result; wherein the target large model adopts a dynamic temperature factor to dynamically adjust each attention head of the target large model. According to the technical scheme provided by the embodiment of the invention, the attention weight of each attention head in the large model can be adaptively adjusted, and the processing capability of the large model on a long text and a complex dependency relationship is enhanced, so that the generalization capability of the large model is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a text processing method, device, equipment, storage and program based on a long context large model improved by a dynamic temperature factor. Background Art

[0002] Context window expansion technology is one of the key innovations in large model technology. It allows large models to process longer input sequences, thereby demonstrating deeper understanding and more accurate output in complex language understanding and generation tasks.

[0003] Currently, position interpolation is mainly used to scale the position encoding to expand the context window of large models. However, position interpolation often shortens the average minimum distance between tokens in the sequence when expanding the context window, resulting in a more concentrated distribution of activation functions in the attention mechanism, thereby limiting the ability of large models to effectively capture long-distance dependencies. Summary of the invention

[0004] The embodiments of the present invention provide a text processing method, apparatus, device, storage and program for a long-context large model improved based on a dynamic temperature factor, which can adaptively adjust the attention weight of each attention head in the large model, enhance the large model's processing capability for long texts and complex dependencies, and thus improve the generalization capability of the large model.

[0005] According to one aspect of the present invention, a text processing method based on a long context large model improved by a dynamic temperature factor is provided, comprising:

[0006] Acquire a target data set; wherein the target data set includes extended length text data;

[0007] Inputting the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result;

[0008] The target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

[0009] According to another aspect of the present invention, there is provided a text processing device based on a long context large model improved by a dynamic temperature factor, comprising:

[0010] A target data set acquisition module, used to acquire a target data set; wherein the target data set includes extended length text data;

[0011] A model processing result acquisition module is used to input the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result;

[0012] The target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text processing method based on the long context large model improved by the dynamic temperature factor as described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text processing method based on the long context large model improved by the dynamic temperature factor as described in any embodiment of the present invention when executed.

[0018] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program, which, when executed by a processor, implements the text processing method based on the long context large model improved by the dynamic temperature factor as described in any embodiment of the present invention.

[0019] The embodiment of the present invention dynamically adjusts the attention heads of the target large model through a dynamic temperature factor. Furthermore, a target data set including extended length text data can be obtained, and then the target data set can be input into the target large model, so that the target data set can be processed by the target large model to obtain a model processing result, thereby solving the problem of poor processing capability of long texts and complex dependencies of existing large models, and being able to adaptively adjust the attention weights of each attention head in the large model, thereby enhancing the large model's processing capability for long texts and complex dependencies, thereby improving the generalization capability of the large model.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 It is a flowchart of a text processing method based on a long context large model improved by a dynamic temperature factor provided in the first embodiment of the present invention;

[0023] Figure 2 It is a flowchart of a text processing method based on a long context large model improved by a dynamic temperature factor provided in the second embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of a text processing device based on a long context large model improved by a dynamic temperature factor provided in a third embodiment of the present invention;

[0025] Figure 4 A schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0027] It should be noted that the terms "target" and "original" in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] Embodiment 1

[0029] Figure 1This is a flowchart of a text processing method based on a long-context large model improved by a dynamic temperature factor, provided in the first embodiment of the present invention. This embodiment is applicable to the case where a large model that dynamically adjusts the attention head using a dynamic temperature factor is used to process text. The method can be executed by a text processing device based on a long-context large model improved by a dynamic temperature factor. The device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. The electronic device can be a terminal device or a server device. As long as the text processing method based on a long-context large model improved by a dynamic temperature factor can be executed, the embodiment of the present invention does not limit the specific device type of the electronic device. Accordingly, Figure 1 As shown, the method includes the following operations:

[0030] S110, obtaining a target data set; wherein the target data set includes extended length text data.

[0031] Among them, the target data set can be a data set to be processed. Exemplarily, the target data set can include but is not limited to extended length text data, etc. The so-called extended length text data can be text content with a longer length relative to conventional length text. Therefore, the extended length text data includes more information, and there are semantic associations across a longer distance between texts. The length of the extended length text data is determined by the characteristics of the target macro model, and the embodiment of the present invention does not limit the length of the extended length text data.

[0032] In an embodiment of the present invention, before using the target large model to perform text processing, a target data set including extended length text data may be firstly acquired as input data of the target large model.

[0033] S120. Input the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result; wherein, the target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

[0034] Among them, the target large model can be any available large language model, which can be used to process the target data set. Large language model (LLM), also called large language model, refers to a deep learning model that can process text sequences using a large amount of relevant data (such as text data, voice data, or text and picture data), which can generate natural language text or understand the meaning of language text. Such models usually have a parameter scale of more than billions. Large language models can handle a variety of natural language tasks, such as text classification, question and answer, and dialogue, and are widely used. The input of the large language model is data, such as text data, voice data, or text and picture data. The large language model can obtain the corresponding word vector representation by encoding the input data, and further decodes the word vector obtained by the encoding, thereby automatically completing the processing of the input data and obtaining the output data corresponding to the input data. Exemplarily, text can be input to the large language model, and the large language model processes and predicts the input text, and outputs the answer text corresponding to the text. Exemplarily, the target large model can be a large model based on a decoder-only architecture in a transformer architecture, for example, an LLaMa2 model with a parameter size of 7B. The dynamic temperature factor can be a parameter for adjusting the output distribution of the attention mechanism. Each attention head of the target large model can be a core component of the multi-head attention mechanism in the target large model, and each attention head can be responsible for processing different subspaces of the input data, thereby capturing different contextual information.

[0035] Accordingly, after obtaining the target data set including the extended length text, the target data set can be input into the target large model, and further, the target data set can be subjected to text processing by the target large model, thereby obtaining the model processing result. It should be noted that before the target data set is processed by the target large model, the attention heads of the target large model can be dynamically adjusted by using the dynamic temperature factor, so that the target large model can distribute attention more evenly, thereby improving the target large model's ability to capture extended length text data information.

[0036] The embodiment of the present invention dynamically adjusts the attention heads of the target large model through a dynamic temperature factor. Furthermore, a target data set including extended length text data can be obtained, and then the target data set can be input into the target large model, so that the target data set can be processed by the target large model to obtain a model processing result, thereby solving the problem of poor processing capability of long texts and complex dependencies of existing large models, and being able to adaptively adjust the attention weights of each attention head in the large model, thereby enhancing the large model's processing capability for long texts and complex dependencies, thereby improving the generalization capability of the large model.

[0037] Embodiment 2

[0038] Figure 2 : is a flowchart of a text processing method based on a long context large model improved by a dynamic temperature factor provided in the second embodiment of the present invention. This embodiment is specific based on the above embodiment. In this embodiment, a variety of specific optional implementation methods are provided before obtaining the target data set. Figure 2 As shown, the method of this embodiment may include:

[0039] S210. Optimize and update the RoPE position encoding of the target large model using the NTK-by-parts position interpolation method.

[0040] Among them, NTK-by-parts (Neural Tangent Kernel-by-parts) position interpolation is usually used to process the position information of sequence data. This method mainly decomposes the position encoding process into several parts for interpolation. RoPE (Rotary Position Embedding) position encoding can be a position encoding technology for processing sequence data.

[0041] In an embodiment of the present invention, before processing the target data set using the target large model, the NTK-by-parts position interpolation method can be used to replace the traditional linear interpolation method to optimize and update the RoPE position encoding of the target large model.

[0042] In an optional embodiment of the present invention, the method of optimizing and updating the RoPE position encoding of the target large model by using the NTK-by-parts position interpolation method may include: optimizing and updating the RoPE position encoding of the target large model by using the NTK-by-parts position interpolation method based on the following formula:

[0043] f NTK-by-parts (x m ,m,θ d )=f(x m ,m,h(θ d ))

[0044]

[0045] Among them, f is the transformation function of RoPE position encoding, f NTK-by-parts The optimized transformation function for NTK-by-parts position interpolation, γ d (r) is the ramp function, h(θ d) is a frequency scaling function, s is a scaling factor, and the scaling factor is determined by the ratio of the length of the extended length text data to the length of the original context. Exemplarily, when the processing capability of the target large model for the extended length text data increases from 4K to 8K, the scaling factor may be 2, L is the original context length, r is the number of rotations under the original context length L, α is the interpolation threshold, and β is the extrapolation threshold. Exemplarily, α may be 1, and β may be 32. The embodiment of the present invention does not limit the specific values ​​of α and β. d is the wavelength of the RoPE position encoding in d dimension, θ d is the rotation angle of the RoPE position encoding in the d dimension, x m is the token at position m, where m is the position index of the context.

[0046] In an embodiment of the present invention, the core of the method of NTK-by-parts position interpolation to optimize and update the RoPE position encoding of the target large model is to adjust the frequency of RoPE position encoding through nonlinear interpolation, so that the target large model can adapt to text data of different lengths. Specifically, NTK-by-parts position interpolation uses the dimension in the target data set whose rotation times exceed the extrapolation threshold as a high-frequency dimension. When processing the high-frequency dimension, NTK-by-parts position interpolation does not interpolate, but directly extrapolates, thereby ensuring sensitivity to rapidly changing features. At the same time, the dimension in the target data set whose rotation times are lower than the interpolation threshold is used as a low-frequency dimension. When processing the low-frequency dimension, interpolation is used for processing. When the number of rotations in the target data set is between the interpolation threshold and the extrapolation threshold, interpolation and extrapolation can be performed simultaneously. The above scheme can effectively reduce the loss of high-frequency information while maintaining the target large model's processing capability for extended-length text data.

[0047] S220. Optimize the activation function of the attention mechanism of the target large model according to the dynamic temperature factor to obtain an optimized target large model.

[0048] Among them, the activation function of the attention mechanism of the target large model can be a softmax (normalized exponential) function.

[0049] Correspondingly, after optimizing and updating the RoPE position encoding of the target large model by using the NTK-by-parts position interpolation method, the activation function of the attention mechanism of the target large model can be optimized according to the dynamic temperature factor, so as to obtain the optimized target large model.

[0050] In a specific example, the initialization matrix of the dynamic temperature factor can be presented as a Gaussian distribution with a mean of 1.2 and a variance of 0.2. A temperature factor slightly greater than 1 can reverse the phenomenon that the average entropy of the attention distribution decreases due to the increase in token density during the interpolation process. At the same time, the Gaussian distribution allows for moderate differences between different attention heads, so that the target large model can adaptively adjust the attention mechanism of each attention head.

[0051] In an optional embodiment of the present invention, optimizing the activation function of the attention mechanism of the target large model according to the dynamic temperature factor may include: optimizing the activation function of the attention mechanism of the target large model according to the dynamic temperature factor based on the following formula:

[0052]

[0053] Among them, Attention is the attention function, softmax is the activation function used to calculate the attention weight, Q is the query, which indicates the information that needs to be paid attention to in the current task, K is the key, which indicates the vector that matches the query, V is the value, which indicates the specific information of each element in the input sequence, D is the set of hidden neurons, T is the dynamic temperature factor, T ij is the temperature factor of the jth attention head of the ith transformer block, represents the N attention heads in each N transformer block.

[0054] S230: performing a model training process on the optimized target large model.

[0055] Correspondingly, after optimizing the activation function of the attention mechanism of the target large model according to the dynamic temperature factor to obtain the optimized target large model, the optimized target large model can be trained, so that the optimized target large model can be fine-tuned.

[0056] In an optional embodiment of the present invention, the model training process for the optimized target large model may include: obtaining a target sample data set; wherein the target sample data set includes data samples of extended length and data samples of original length; inputting the target sample data set into the optimized target large model, so as to perform text processing on the target sample data set through the optimized target large model to obtain a model training result.

[0057] The target sample data set may be a sample data set used to train the optimized target large model. The data sample of extended length may be a text content with a longer length in the target sample data set relative to the data sample of original length. Therefore, the data sample of extended length may include more information, and there are semantic associations between texts spanning a longer distance. The data sample of original length may be a data sample in the initial collection or generation state without any length change operation. The model training result may be the output and evaluation index obtained after the model training is completed.

[0058] In an embodiment of the present invention, when the optimized target large model is trained, a target sample data set including data samples of extended length and data samples of original length can be first obtained, and further, the target sample data set can be input into the optimized target large model, so that the target sample data set can be processed by the optimized target large model to obtain a model training result. In the above scheme, the target sample data set includes both data samples of extended length and data samples of original length, which can help the target large model learn long-distance dependencies while maintaining the processing capability of short-distance sequences.

[0059] In an optional embodiment of the present invention, the model training process for the optimized target large model may also include: in the process of model training for the optimized target large model, learning and updating the network parameters and the dynamic temperature factor of the target large model through a back propagation algorithm; repeatedly training the optimized target large model until it is determined that the optimized target large model meets the training termination conditions.

[0060] Among them, the back propagation algorithm can be an algorithm used to learn the network parameters and dynamic temperature factors of the target large model during the model training process. The network parameters can be all the trainable weights and biases that constitute the target large model. The training termination condition can be a pre-set performance indicator for stopping the training of the target large model. Exemplarily, when the target large model reaches or exceeds the corresponding indicator during the training process, the training terminates; or when the maximum number of training times is reached, the training terminates. Exemplarily, the performance indicators may include but are not limited to perplexity and authenticity, etc., and the embodiments of the present invention do not limit the specific types of performance indicators included. Generally, the lower the perplexity and the higher the authenticity, the better the performance of the target large model.

[0061] In the embodiment of the present invention, during the model training process of the optimized target large model, the network parameters and dynamic temperature factors of the target large model can also be updated through the back propagation algorithm, and the optimized target large model can be repeatedly trained until it is determined that the optimized target large model meets the training termination condition. The above scheme can improve the generalization ability of the target large model by fine-tuning the network parameters and dynamic temperature factors of the target large model.

[0062] The embodiment of the present invention optimizes and updates the RoPE position encoding of the target large model by adopting the NTK-by-parts position interpolation method. Furthermore, the activation function of the attention mechanism of the target large model can be optimized according to the dynamic temperature factor to obtain the optimized target large model, so that the model training process can be performed on the optimized target large model to solve the problem of poor processing capability of long texts and complex dependencies of the existing large model. The attention weights of each attention head in the large model can be adaptively adjusted to enhance the processing capability of the large model for long texts and complex dependencies, thereby improving the generalization capability of the large model.

[0063] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0064] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

[0065] Embodiment 3

[0066] Figure 3 is a schematic diagram of a text processing device based on a long context large model improved by a dynamic temperature factor provided in Embodiment 3 of the present invention, such as Figure 3 As shown, the device includes: a target data set acquisition module 310 and a model processing result acquisition module 320, wherein:

[0067] The target data set acquisition module 310 is used to acquire a target data set; wherein the target data set includes extended length text data.

[0068] A model processing result acquisition module 320 is used to input the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result;

[0069] The target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

[0070] The embodiment of the present invention dynamically adjusts the attention heads of the target large model through a dynamic temperature factor. Furthermore, a target data set including extended length text data can be obtained, and then the target data set can be input into the target large model, so that the target data set can be processed by the target large model to obtain a model processing result, thereby solving the problem of poor processing capability of long texts and complex dependencies of existing large models, and being able to adaptively adjust the attention weights of each attention head in the large model, thereby enhancing the large model's processing capability for long texts and complex dependencies, thereby improving the generalization capability of the large model.

[0071] Optionally, the above-mentioned device may also include a model optimization module, which is specifically used to: optimize and update the RoPE position encoding of the target large model using the NTK-by-parts position interpolation method; optimize the activation function of the attention mechanism of the target large model according to the dynamic temperature factor to obtain an optimized target large model; and perform a model training process on the optimized target large model.

[0072] Optionally, the model optimization module is further used to: optimize and update the RoPE position encoding of the target large model by using the NTK-by-parts position interpolation method based on the following formula:

[0073] f NTK-by-parts (x m ,m,θ d )=f(x m ,m,h(θ d ))

[0074]

[0075] Among them, f is the transformation function of RoPE position encoding, f NTK-by-parts The optimized transformation function for NTK-by-parts position interpolation, γ d (r) is the ramp function, h(θ d ) is the frequency scaling function, s is the scaling factor, L is the original context length, r is the number of rotations under the original context length L, α is the interpolation threshold, β is the extrapolation threshold, λ d is the wavelength of the RoPE position encoding in d dimension, θ d is the rotation angle of the RoPE position encoding in the d dimension, x m is the token at position m, where m is the position index of the context.

[0076] Optionally, the model optimization module is further used to optimize the activation function of the attention mechanism of the target large model according to the dynamic temperature factor based on the following formula:

[0077]

[0078] Among them, Attention is the attention function, softmax is the activation function used to calculate the attention weight, Q is the query, which indicates the information that needs to be paid attention to in the current task, K is the key, which indicates the vector that matches the query, V is the value, which indicates the specific information of each element in the input sequence, D is the set of hidden neurons, T is the dynamic temperature factor, T ij is the temperature factor of the jth attention head of the ith transformer block, represents the N attention heads in each N transformer block.

[0079] Optionally, the model optimization module is also used to: obtain a target sample data set; wherein the target sample data set includes data samples of extended length and data samples of original length; input the target sample data set into the optimized target large model, so as to perform text processing on the target sample data set through the optimized target large model to obtain a model training result.

[0080] Optionally, the model optimization module is also used to: in the process of model training of the optimized target large model, learn and update the network parameters and the dynamic temperature factor of the target large model through the back propagation algorithm; repeatedly train the optimized target large model until it is determined that the optimized target large model meets the training termination conditions.

[0081] The above-mentioned text processing device based on the long context large model improved by the dynamic temperature factor can execute the text processing method based on the long context large model improved by the dynamic temperature factor provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the text processing method based on the long context large model improved by the dynamic temperature factor provided in any embodiment of the present invention.

[0082] Since the above-mentioned text processing device based on the long context large model improved by the dynamic temperature factor is a device that can execute the text processing method based on the long context large model improved by the dynamic temperature factor in the embodiment of the present invention, based on the text processing method based on the long context large model improved by the dynamic temperature factor introduced in the embodiment of the present invention, the technical personnel of the field can understand the specific implementation of the text processing device based on the long context large model improved by the dynamic temperature factor in this embodiment and its various variations, so how the text processing device based on the long context large model improved by the dynamic temperature factor implements the text processing method based on the long context large model improved by the dynamic temperature factor in the embodiment of the present invention is not introduced in detail. As long as the technical personnel of the field implement the device adopted by the text processing method based on the long context large model improved by the dynamic temperature factor in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0083] Embodiment 4

[0084] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0085] like Figure 4 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0086] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0087] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the text processing method of the long context large model improved based on the dynamic temperature factor.

[0088] In some embodiments, the text processing method based on the long context large model improved by the dynamic temperature factor can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the text processing method based on the long context large model improved by the dynamic temperature factor described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to execute the text processing method based on the long context large model improved by the dynamic temperature factor in any other appropriate manner (for example, by means of firmware).

[0089] Optionally, a text processing method of a long-context large model improved based on a dynamic temperature factor may include: obtaining a target data set; wherein the target data set includes extended-length text data; inputting the target data set into a target large model to perform text processing on the target data set through the target large model to obtain a model processing result; wherein the target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

[0090] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0092] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0093] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0094] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0095] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0096] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0097] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A text processing method based on a long context large model improved by a dynamic temperature factor, characterized in that: include: Acquire a target data set; wherein the target data set includes extended length text data; Inputting the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result; The target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

2. The method according to claim 1, characterized in that Before obtaining the target data set, the method further includes: The rotational position embedding RoPE position encoding of the target large model is optimized and updated by using a segmented neural tangent kernel NTK-by-parts position interpolation method; Optimizing the activation function of the attention mechanism of the target large model according to the dynamic temperature factor to obtain an optimized target large model; A model training process is performed on the optimized target large model.

3. The method according to claim 2, characterized in that The method of using NTK-by-parts position interpolation to optimize and update the RoPE position encoding of the target large model includes: The RoPE position encoding of the target large model is optimized and updated using the NTK-by-parts position interpolation method based on the following formula: f NTK-by-parts (x m ,m,θ d )=f(x m ,m,h(θ d )) Where f is the transformation function of RoPE position encoding, f NTK-by-parts The optimized transformation function for NTK-by-parts position interpolation, γ d (r) is the ramp function, h(θ d ) is the frequency scaling function, s is the scaling factor, L is the original context length, r is the number of rotations under the original context length L, α is the interpolation threshold, β is the extrapolation threshold, λ d is the wavelength of the RoPE position encoding in d dimension, θ d is the rotation angle of the RoPE position encoding in the d dimension, x m is the word token at position m, and m is the position index of the context.

4. The method according to claim 2, characterized in that: The step of optimizing the activation function of the attention mechanism of the target large model according to the dynamic temperature factor includes: The activation function of the attention mechanism of the target large model is optimized according to the dynamic temperature factor based on the following formula: Among them, Attention is the attention function, softmax is the activation function used to calculate the attention weight, Q is the query, which indicates the information that needs to be paid attention to in the current task, K is the key, which indicates the vector that matches the query, V is the value, which indicates the specific information of each element in the input sequence, D is the set of hidden neurons, T is the dynamic temperature factor, T ij is the temperature factor of the jth attention head of the ith transformer block, represents the N attention heads in each N transformer block.

5. The method according to claim 4, characterized in that The process of performing model training on the optimized target large model includes: Acquire a target sample data set; wherein the target sample data set includes data samples of extended length and data samples of original length; The target sample data set is input into the optimized target large model, so as to perform text processing on the target sample data set through the optimized target large model to obtain a model training result.

6. The method according to claim 5, characterized in that Also includes: In the process of model training the optimized target large model, the network parameters of the target large model and the dynamic temperature factor are updated by back propagation algorithm learning; Repeat training the optimized target large model until it is determined that the optimized target large model meets the training termination condition.

7. A text processing device based on a long context large model improved by a dynamic temperature factor, characterized in that: include: A target data set acquisition module, used to acquire a target data set; wherein the target data set includes extended length text data; A model processing result acquisition module is used to input the target data set into the target large model, so as to perform text processing on the target data set through the target large model to obtain a model processing result; The target large model uses a dynamic temperature factor to dynamically adjust each attention head of the target large model.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the text processing method based on the long context large model improved by the dynamic temperature factor as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the text processing method based on a long context large model improved by a dynamic temperature factor as described in any one of claims 1-6 when executed.

10. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the text processing method based on a long context large model improved by a dynamic temperature factor according to any one of claims 1 to 6 is implemented.

Images