A context length expansion method and system for a power supply service command large model

Abstract

The application relates to a context length expansion method and system for a power supply service command large model, which comprises the following steps: adding position coding to input word embedding by using a sine-cosine function, so that the large model considers word sequence information; in a multi-head self-attention mechanism, rotating and transforming the coding vector of each position, using the position information after the rotation and transformation to calculate attention scores, and better capturing long-distance dependence; interpolating the rotated position coding by using the neural kernel characteristic, fine-tuning the semantic large model by using a plurality of data, expanding the context window, and reducing high-frequency information loss; constructing a power supply service instruction data set, checking and correcting the instruction data set by experts, fine-tuning the improved semantic large model by using the power supply service instruction data set, and making the semantic large model adapt to the expanded context length.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and more specifically, to a method and system for extending the context length of a large-scale power supply service command model. Background Technology

[0002] In modern power systems, power supply service command involves a large amount of data processing and decision-making. With the development of artificial intelligence technology, especially the application of large-scale models, the intelligence level of power supply service command systems has been significantly improved. However, current large-scale models have certain limitations in handling long contexts, especially in complex power supply service command scenarios, where they cannot effectively process and utilize long-term series data, thus limiting their application effectiveness.

[0003] The main challenge lies in the sheer volume of information contained in long texts, demanding highly accurate understanding and processing capabilities from the model. However, due to the limited context length of the model, longer texts exceed its processing capacity, leading to information loss or errors. Furthermore, processing long texts introduces a significant computational and storage burden, severely testing the model's efficiency and performance. One current mainstream approach employs a sliding window method to actively discard preceding text, retaining only the most recent input or only a portion of the input. However, this results in the loss of historical contextual information, failing to comprehensively integrate full-text information to fully answer questions or generate content. In addition, some innovative methods adapt to long contexts by improving the model's positional encoding, scaling the positional information of rotational positional encoding to the required length.

[0004] In complex business scenarios, the additional manpower, time, and memory resources required for retraining the model when integrating knowledge bases and prompts are excessive. Therefore, improving the context length handling capability of large models, allowing for the expansion of the model's context window to the required length without fine-tuning while sacrificing a small amount of model performance, and enhancing its application in power supply service command, has become a key requirement for current technological development.

[0005] Existing technologies, such as Chinese patent number CN118427332A, disclose a method for fine-tuning the context expansion of a large-scale food safety model based on a text paraphrasing mechanism. This method includes establishing reference documents, forming groups from multiple reference documents, designing question-and-answer pairs from the first reference document, and providing relevant original text in the answers to the question-and-answer pairs; shuffling the order of the reference documents within the group to form a list of reference documents and instruction fine-tuning data; establishing a language model, importing the instruction fine-tuning data into the language model, and training the fine-tuning to obtain a long-text language model. This application effectively addresses the problems of low question-and-answer accuracy and neglect of contextual information in existing language models when dealing with long texts, significantly improving the accuracy of question-and-answering for long texts, and avoiding the significant increase in cost caused by manual annotation.

[0006] The problem with the above-mentioned existing technology is:

[0007] 1. Reliance on specific reference documents: If the quality of the reference documents is poor, the coverage is incomplete, or there are biases, it may affect the performance and accuracy of the final model. Designing and training solely based on existing reference documents may not be able to adapt to new food safety-related issues and situations not included in the reference documents.

[0008] 2. While this method aims to address the problem of long texts, its adaptability may be limited for long texts across different domains or types. The language structure and semantic relationships within different domains can vary significantly, leading to poor performance of the model when applied across different domains. Summary of the Invention

[0009] To address the aforementioned technical problems, this invention proposes a method and system for extending the context length of a large-scale power supply service command model.

[0010] The technical solution of this invention is as follows:

[0011] This invention proposes a method for extending the context length of a large-scale power supply service command model, comprising the following steps:

[0012] Step S1: Use sine and cosine functions to add positional encoding to the input word embedding, so that the large model can take word order information into account;

[0013] Step S2: In the multi-head self-attention mechanism, the encoding vector at each position is rotated and transformed, and the attention score is calculated using the position information after rotation and transformation. Long-distance dependencies are captured by introducing position information.

[0014] Step S3: Interpolate the position encoding after rotation transformation using the characteristics of neural nuclei, fine-tune the semantic model using several data, expand the context window, and reduce the loss of high-frequency information;

[0015] Step S4: Construct a power supply service instruction dataset. Experts review, revise, and expand the power supply service instruction dataset. Use the power supply service instruction dataset to fine-tune the improved semantic big model, making the semantic big model adapt to the expanded context length.

[0016] In a preferred embodiment, the positional encoding is added to the input word embedding using sine and cosine functions, and the specific formula for representing the positional information is as follows:

[0017]

[0018] in, It is the absolute position of the word in the sequence. It is the dimension index in the word vector. It refers to the dimension of the model; odd-numbered dimensions are used. Function computation, even-numbered dimensions use Function calculation; the above-mentioned sine and cosine positional encoding calculation method is used to calculate the relative positions of surrounding words so that the large model can learn the relative positions of surrounding words, because for any distance l, Can be represented as A linear function.

[0019] In a preferred embodiment, the process of calculating the attention score using the position information after rotational transformation is as follows:

[0020] For a given length The input sequence, and its corresponding embedding representation, are as follows:

[0021]

[0022] in Indicates the first Each word element corresponds to Word embedding vector;

[0023] For the location in vector The encoding method for its rotation position is represented as follows:

[0024]

[0025] in, Let be the rotation angle of vector x in dimension d;

[0026] Query vector for each lexical position and key vector The elements of the vectors are rotated in pairs, and the real parts are then used to calculate the inner product between the query and key vectors to obtain the self-attention result. The specific calculation formula is as follows:

[0027]

[0028] In the formula, Re represents taking the real part of the function; I( () represents the inner product calculation function;

[0029] Expanding the above equation, we get:

[0030]

[0031] In the formula, j = 1, 2, 3, .

[0032] In a preferred embodiment, the interpolation of the position code after rotational transformation using the characteristics of neural nuclei specifically includes the following steps:

[0033] Define a scaling factor To extend the context length Compared with the original context length The ratio between:

[0034]

[0035] Further vector The rotation position encoding function is rewritten as about the input vector ,Location and angle Functions:

[0036]

[0037] In addition, Defined as the first The wavelength embedded at the rotation position in each hidden dimension represents the wavelength of the complete rotation. Required lexical length:

[0038]

[0039] in, D; This is a constant; for a given neural network model and input data, the neural kernel matrix is ​​calculated, reflecting the similarity between the input data and the model's gradient information on these data, specifically manifested as... Function improvements:

[0040]

[0041]

[0042]

[0043] In computational self-attention mechanisms Previously in The parameter v introduced above has a uniform effect on the model's text understanding ability, and the length scaling technique is used to further adjust the text position encoding length; the specific calculation of its attention weights is modified as follows:

[0044] .

[0045] On the other hand, the present invention also provides a context length extension system for a large-scale power supply service command model, comprising:

[0046] The positional encoding module uses sine and cosine functions to add positional encoding to the input word embedding, enabling the large model to consider word order information;

[0047] The rotation transformation module rotates the encoded vector at each position in the multi-head self-attention mechanism, calculates the attention score using the position information after rotation transformation, and captures long-distance dependencies by introducing position information.

[0048] The interpolation fine-tuning module uses the characteristics of neural nuclei to interpolate the positional encoding after rotation transformation, uses several data to fine-tune the semantic large model, expands the context window, and reduces the loss of high-frequency information.

[0049] The instruction fine-tuning module constructs a power supply service instruction dataset, which is then reviewed, corrected, and expanded by experts. This dataset is then used to fine-tune the improved semantic model, adapting it to the expanded context length.

[0050] In a preferred embodiment, the position encoding module uses sine and cosine functions to add position encoding to the input word embedding. The specific formula for representing position information is as follows:

[0051]

[0052] in, It is the absolute position of the word in the sequence. It is the dimension index in the word vector. It refers to the dimension of the model; odd-numbered dimensions are used. Function computation, even-numbered dimensions use Function calculation; the above-mentioned sine and cosine positional encoding calculation method is used to calculate the relative positions of surrounding words so that the large model can learn the relative positions of surrounding words, because for any distance l, Can be represented as A linear function.

[0053] In a preferred embodiment, the rotation transformation module uses the position information after rotation transformation to calculate the attention score, and the specific process is as follows:

[0054] For a given length The input sequence, and its corresponding embedding representation, are as follows:

[0055]

[0056] in Indicates the first Each word element corresponds to Word embedding vector;

[0057] For the location in vector The encoding method for its rotation position is represented as follows:

[0058]

[0059] in, Let x be the rotation angle of vector x in dimension d;

[0060] Query vector for each lexical position and key vector The elements of the vectors are rotated in pairs, and the real parts are then used to calculate the inner product between the query and key vectors to obtain the self-attention result. The specific formula for calculating the attention score is as follows:

[0061]

[0062] In the formula, Re represents taking the real part of the function; I( () represents the inner product calculation function;

[0063] Expanding the above equation, we get:

[0064]

[0065] In the formula, j = 1, 2, 3, ;

[0066] Self-attention score Relying solely on the relative position rt, here and These are the query and key vectors for a specific attention head. At each layer, rotational position encoding is applied to the query and key embeddings to compute the attention score.

[0067] In a preferred embodiment, the interpolation fine-tuning module uses the characteristics of neural nuclei to interpolate the position code after rotation transformation, specifically including the following steps:

[0068] Define a scaling factor To extend the context length Compared with the original context length The ratio between:

[0069]

[0070] Further vector The rotation position encoding function is rewritten as about the input vector ,Location and angle Functions:

[0071]

[0072] In addition, Defined as the first The wavelength embedded at the rotation position in each hidden dimension represents the wavelength of the complete rotation. Required lexical length:

[0073]

[0074] in, D; This is a constant; for a given neural network model and input data, the neural kernel matrix is ​​calculated, reflecting the similarity between the input data and the model's gradient information on these data, specifically manifested as... Function improvements:

[0075]

[0076]

[0077]

[0078] In computational self-attention mechanisms Previously in The parameter v introduced above has a uniform effect on the model's text understanding ability, and the length scaling technique is used to further flexibly adjust the text position encoding length; the specific calculation of its attention weights is modified as follows:

[0079] .

[0080] In another aspect, the present invention also provides an electronic device having a computer program stored thereon, which, when executed by a processor, implements a context length extension method for a power supply service command big model as described in any embodiment of the present invention.

[0081] In another aspect, the present invention also provides a computer-readable medium for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement a context length extension method for a power supply service command big model as described in any embodiment of the present invention.

[0082] The present invention has the following beneficial effects:

[0083] 1. Improved long text processing capabilities: By improving rotational position encoding, the model is able to better capture long-distance dependencies and can be effectively used in multi-head self-attention mechanisms, solving the problem of limited context length in existing technologies.

[0084] 2. Improved Model Performance and Efficiency: This invention extends the context length through interpolation, reducing the loss of high-frequency information without requiring additional model fine-tuning, thus lowering the computational burden. It can retain more dialogue history information during the knowledge service process of the power supply service command model, improving the fluency and accuracy of responses, thereby enhancing the overall intelligence and responsiveness of the system in complex power supply service command scenarios. Attached Figure Description

[0085] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0086] Figure 1 This is a schematic diagram of the method flow in Example 1;

[0087] Figure 2 Flowchart for calculating attention scores for rotational position encoding;

[0088] Figure 3 A flowchart illustrating the method for fine-tuning instructions in the power customer service field. Detailed Implementation

[0089] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0090] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0091] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0092] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0093] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0094] Example 1:

[0095] To make the objectives, technical solutions, and advantages of this invention clearer, specific embodiments of this application will be described below, with reference to the accompanying drawings. Figure 1 The technical solution of the present invention will be clearly and completely described.

[0096] To address the problems of existing technologies, this invention provides a method for extending the context length of a large-scale power supply service command model, comprising the following steps:

[0097] Step S1: Use sine and cosine functions to add positional encoding to the input word embedding, so that the large model can take word order information into account;

[0098] The semantic large-scale model is based on the Transformer architecture, mainly consisting of an embedding layer and multiple decoders. It does not contain recursion or convolution, and its parallel input ignores word order information. To allow the model to consider the order of words in a sentence, standard positional encoding uses sine and cosine functions to directly add positional codes to the input word embeddings to represent the positional information of the sequence.

[0099]

[0100] in, It is the absolute position of the word in the sequence. It is the dimension index in the word vector. It refers to the dimension of the model; odd-numbered dimensions are used. Function computation, even-numbered dimensions use Function computation; this method of computation allows large models to learn the relative positions of surrounding words because for any distance l, It can be represented as A linear function.

[0101] Step S2: In the multi-head self-attention mechanism, the encoding vector at each position is rotated and transformed, and the attention score is calculated using the position information after rotation and transformation. Long-distance dependencies are captured by introducing position information.

[0102] The core idea of ​​this rotational positional encoding is to rotate the encoded vector at each position to incorporate positional information. Specifically, for each input vector, it is treated as a complex number, and positional information is embedded into it through a rotation operation. This method can better capture long-range dependencies and can be effectively used in multi-head self-attention mechanisms.

[0103] For a given length The input sequence, and its corresponding embedding representation, are as follows:

[0104]

[0105] in Indicates the first Each word element corresponds to Word embedding vector;

[0106] For the location in vector This encoding method can be represented as:

[0107]

[0108] in, Let x be the rotation angle of vector x in dimension d;

[0109] According to Euler's formula:

[0110]

[0111] Where c represents any real number, Let be the base of the natural logarithm, and 'a' be the imaginary unit in complex numbers; then the above exponential function can be expressed as having a real part of . The imaginary part is A complex number;

[0112] Therefore, assuming vector For a two-dimensional vector, with respect to position Define the rotation matrix :

[0113]

[0114] Then, for that given length The input sequence is rotated before self-attention computation. For each even index u and odd index u+1 in the matrix, apply a rotation matrix. :

[0115]

[0116] This is equivalent to pairing even-numbered and odd-numbered dimensions for a two-dimensional rotation. Finally, the rotation vectors for all positions are concatenated to obtain the final rotation position code.

[0117] Query vector for each lexical position and key vector The elements of the vectors are rotated in pairs, and the real parts are then used to calculate the inner product between the query and key vectors to obtain the self-attention result. The specific formula for calculating the attention score is as follows:

[0118]

[0119] In the formula, Re represents taking the real part of the function; I( () represents the inner product calculation function;

[0120] Expanding the above equation, we get:

[0121]

[0122] In the formula, j = 1, 2, 3, ;

[0123] Self-attention score Relying solely on the relative position rt, here and These are the query and key vectors for a specific attention head. At each layer, rotational position encoding is applied to the query and key embeddings to compute the attention score.

[0124] Step S3: Interpolate the rotation position encoding using the characteristics of the neural nucleus, fine-tune the semantic model using several data, expand the context window, and reduce the loss of high-frequency information;

[0125] Semantic large-scale models are pre-trained with a fixed context length, making it difficult to directly extend the context window defined during training for positional encoding. To address the need for context length extension in power supply service command, this paper utilizes neural kernel characteristics to interpolate rotational positional encoding and fine-tunes it with a relatively small amount of data to extend the context window of the power supply service command large-scale model. The specific steps are as follows:

[0126] First, define a scaling factor. To extend the context length Compared with the original context length The ratio between:

[0127]

[0128] Further, in step 2, for vectors The encoding function is rewritten as about the input vector ,Location and rotation angle Functions:

[0129]

[0130] In addition, Defined as the first The wavelength embedded at the rotation position in each hidden dimension represents the wavelength of the complete rotation. Required lexical length:

[0131]

[0132] in, D; This is a constant; for a given neural network model and input data, the neural kernel matrix is ​​calculated, reflecting the similarity between the input data and the model's gradient information on these data, specifically manifested as... Function improvements:

[0133]

[0134]

[0135]

[0136] By using the above formula, different frequency dimensions of the rotation position encoding are scaled to varying degrees to reduce high frequencies and increase low frequencies.

[0137] In computational self-attention mechanisms Previously in The parameter v introduced above has a uniform effect on the model's text understanding ability, but the length of the text position encoding can be further adjusted flexibly using length scaling techniques; the specific calculation of its attention weights is modified as follows:

[0138]

[0139] By simply scaling the complex rotational position encoding by the same proportion, and Both are constant factors Scaling can effectively alter the attention mechanism without requiring additional training.

[0140] Step S4: Construct a power supply service instruction dataset. Experts review, revise, and expand the instruction dataset. Use the power supply service instruction dataset to fine-tune the improved semantic model, making the semantic model adapt to the expanded context length.

[0141] A method for fine-tuning instructions in the power supply service field is designed. First, actual knowledge service materials from the system are collected and cleaned and structured. Then, the business logic of the power supply service knowledge service is constructed, forming a power supply service instruction dataset in the form of (instruction, input, output) triples. To construct a high-quality and diverse instruction dataset, power supply command business experts are invited to check and correct erroneous answers in the instruction dataset and rewrite instruction sentences to expand the form of instructions in the dataset. After the experts have corrected and expanded the instruction data, the power supply service domain instruction dataset is obtained. During the model training phase, the power supply service command model, improved according to step 3, is fine-tuned on the power supply service domain instruction dataset to make the large model more adaptable to the expanded context length and able to execute corresponding tasks according to the instructions.

[0142] Example 2:

[0143] This embodiment provides a context length extension system for a large-scale power supply service command model, including:

[0144] The positional encoding module uses sine and cosine functions to add positional encoding to the input word embedding, enabling the large model to consider word order information;

[0145] The rotation transformation module rotates the encoded vector at each position in the multi-head self-attention mechanism and uses the position information after rotation transformation to calculate the attention score, which can better capture long-distance dependencies.

[0146] The interpolation fine-tuning module uses the characteristics of neural nuclei to interpolate the rotation position encoding, uses several data to fine-tune the semantic large model, expands the context window, and reduces the loss of high-frequency information.

[0147] The instruction fine-tuning module constructs a power supply service instruction dataset, which is then reviewed, corrected, and expanded by experts. The improved semantic model is then fine-tuned using the power supply service instruction dataset to adapt the semantic model to the expanded context length.

[0148] Example 3:

[0149] This embodiment provides an electronic device on which a computer program is stored. When the computer program is executed by a processor, it implements a context length expansion method for a power supply service command model as described in any embodiment of the present invention.

[0150] Example 4:

[0151] This embodiment provides a computer-readable medium for storing one or more programs that, when executed by one or more processors, cause the one or more processors to implement a context length extension method for a power supply service command big model as described in any embodiment of the present invention.

[0152] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0153] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0154] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0155] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for extending the context length of a large-scale power supply service command model, characterized in that, Includes the following steps: Step S1: Add positional encoding to the input word embedding using sine and cosine functions to enable the large model to consider word order information; the specific formula for representing positional information by adding positional encoding to the input word embedding using sine and cosine functions is as follows: in, It is the absolute position of the word in the sequence. It is the dimension index in the word vector. It refers to the dimension of the model; odd-numbered dimensions are used. Function computation, even-numbered dimensions use Function calculation; the above-mentioned sine and cosine positional encoding calculation method is used to calculate the relative positions of surrounding words so that the large model can learn the relative positions of surrounding words, because for any distance l, Can be represented as A linear function; Step S2: In the multi-head self-attention mechanism, the encoding vector at each position is rotated, and the attention score is calculated using the positional information after rotation. This captures long-distance dependencies by introducing positional information. The specific process for calculating the attention score using the positional information after rotation is as follows: For a given length The input sequence, and its corresponding embedding representation, are as follows: in Indicates the first Each word element corresponds to Word embedding vector; For the location in vector The encoding method for its rotation position is represented as follows: in, Let be the rotation angle of vector x in dimension d; Query vector for each lexical position and key vector The elements of the vectors are rotated in pairs, and the real parts are then used to calculate the inner product between the query and key vectors to obtain the self-attention result. The specific calculation formula is as follows: In the formula, Re represents taking the real part of the function; I( () represents the inner product calculation function; Expanding the above equation, we get: In the formula, j = 1, 2, 3, ; Step S3: Interpolate the positional encoding after rotation transformation using the characteristics of neural nuclei, fine-tune the semantic model using several data points, expand the context window, and reduce the loss of high-frequency information; the interpolation of the positional encoding after rotation transformation using the characteristics of neural nuclei specifically includes the following steps: Define a scaling factor To extend the context length Compared with the original context length The ratio between: Further vector The rotation position encoding function is rewritten as about the input vector ,Location and angle Functions: In addition, Defined as the first The wavelength embedded at the rotation position in each hidden dimension represents the wavelength of the complete rotation. Required lexical length: in, D; This is a constant; for a given neural network model and input data, the neural kernel matrix is ​​calculated, reflecting the similarity between the input data and the model's gradient information on these data, specifically manifested as... Function improvements: In computational self-attention mechanisms Previously, in The parameter v introduced above has a uniform effect on the model's text understanding ability, and the length scaling technique is used to further adjust the text position encoding length; the specific calculation of its attention weights is modified as follows: Step S4: Construct a power supply service instruction dataset. Experts review, revise, and expand the power supply service instruction dataset. Use the power supply service instruction dataset to fine-tune the improved semantic big model, making the semantic big model adapt to the expanded context length.

2. A context length extension system for a large-scale power supply service command model, characterized in that, include: The positional encoding module uses sine and cosine functions to add positional encoding to the input word embedding, enabling the large model to consider word order information; The position encoding module uses sine and cosine functions to add position encoding to the input word embedding. The specific formula for representing position information is as follows: in, It is the absolute position of the word in the sequence. It is the dimension index in the word vector. It refers to the dimension of the model; odd-numbered dimensions are used. Function computation, even-numbered dimensions use Function calculation; the above-mentioned sine and cosine positional encoding calculation method is used to calculate the relative positions of surrounding words so that the large model can learn the relative positions of surrounding words, because for any distance l, Can be represented as A linear function; The rotation transformation module, in the multi-head self-attention mechanism, rotates the encoded vector at each position and uses the rotated position information to calculate the attention score, thus capturing long-distance dependencies by introducing position information. The specific process of the rotation transformation module calculating the attention score using the rotated position information is as follows: For a given length The input sequence, and its corresponding embedding representation, are as follows: in Indicates the first Each word element corresponds to Word embedding vector; For the location in vector The encoding method for its rotation position is represented as follows: in, Let be the rotation angle of vector x in dimension d; Query vector for each lexical position and key vector The elements of the vectors are rotated in pairs, and the real parts are used to calculate the inner product between the query and key vectors to obtain the self-attention result. The specific formula for calculating the attention score is as follows: In the formula, Re represents taking the real part of the function; I( () represents the inner product calculation function; Expanding the above equation, we get: In the formula, j = 1, 2, 3, ; Self-attention score Relying solely on the relative position rt, here and These are the query and key vectors for a specific attention head. At each layer, rotational position encoding is applied to the query and key embeddings to compute the attention score. The interpolation fine-tuning module utilizes the characteristics of neural kernel division to interpolate the positional encoding after rotation transformation, and uses several data points to fine-tune the semantic model, expanding the context window and reducing the loss of high-frequency information. The interpolation fine-tuning module utilizes the characteristics of neural kernel division to interpolate the positional encoding after rotation transformation, specifically including the following steps: Define a scaling factor To extend the context length Compared with the original context length The ratio between: Further vector The rotation position encoding function is rewritten as about the input vector ,Location and angle Functions: In addition, Defined as the first The wavelength embedded at the rotation position in each hidden dimension represents the wavelength of the complete rotation. Required lexical length: in, D; This is a constant; for a given neural network model and input data, the neural kernel matrix is ​​calculated, reflecting the similarity between the input data and the model's gradient information on these data, specifically manifested as... Function improvements: In computational self-attention mechanisms Previously in The parameter v introduced above has a uniform effect on the model's text understanding ability, and the length scaling technique is used to further flexibly adjust the text position encoding length; the specific calculation of its attention weights is modified as follows: The instruction fine-tuning module constructs a power supply service instruction dataset, which is then reviewed, corrected, and expanded by experts. This dataset is then used to fine-tune the improved semantic model, adapting it to the expanded context length.

3. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the context length expansion method for a power supply service command model as described in claim 1.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the context length expansion method for a power supply service command model as described in claim 1.

Images