Context length extension method and system for large power supply service command model

By adopting cosine function and rotation transformation position coding in the large model, combining neural nucleotomy characteristics and instruction fine-tuning, the information loss and calculation burden problems in long context processing are solved, and the application effect of the model in power supply service command is improved. Cross-domain adaptability.

CN118966282AActive Publication Date: 2024-11-15STATE GRID FUJIAN ELECTRIC POWER RES INST +3
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411102526.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-15
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

Existing large models have information loss and computational burden when dealing with long contexts, and are difficult to adapt to long texts in different fields, resulting in low Q&A accuracy and poor cross-domain application.

Method used

The cosine function is used to add position encoding, rotate and transform position information, interpolate using the neural nucleotomy characteristics, expand the context window, and fine-tune the model through the expert-corrected power supply service instruction data set to adapt to the extended context length.

Benefits of technology

It improves long text processing capabilities, reduces high-frequency information loss, improves model performance and efficiency, can better capture long-distance dependencies, adapt to complex power supply service command scenarios, and improves the fluency and accuracy of reply.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118966282A_ABST
    Figure CN118966282A_ABST
Patent Text Reader

Abstract

The invention relates to a power supply service command large model-oriented context length extension method and system, and the method comprises the following steps: adding a position code into an input word embedding through employing a sine and cosine function, and enabling a large model to consider word order information; in the multi-head self-attention mechanism, rotation transformation is carried out on the coding vector of each position, the attention score is calculated by using the position information after rotation transformation, and the long-distance dependency relationship is better captured; a rotation position code is interpolated by using a nerve kernel cutting characteristic, a semantic large model is finely adjusted by using a plurality of data, a context window is expanded, and high-frequency information loss is reduced; and a power supply service instruction data set is constructed, an expert checks, corrects and expands the instruction data set, and the power supply service instruction data set is used to carry out instruction fine tuning on the improved large semantic model, so that the large semantic model adapts to the expanded context length.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of natural language processing, and more specifically, to a context length extension method and system for a large power supply service command model. Background Art

[0002] In modern power systems, power supply service command involves a large amount of data processing and decision-making processes. With the development of artificial intelligence technology, especially the application of large models, the intelligence level of power supply service command systems has been significantly improved. However, the current large models have certain limitations in processing long contexts, especially in complex power supply service command scenarios, and cannot effectively process and utilize long time series data, which limits their application effect.

[0003] The main challenge is that the amount of information contained in long texts is huge, requiring the model to have the ability to understand and process it with high accuracy. However, due to the limited context length of the model, longer texts will exceed its processing range, resulting in information loss or errors in processing. In addition, processing long texts will also introduce huge computational and storage burdens, which will severely test the efficiency and performance of the model. One of the current mainstream methods is to use a sliding window method to actively discard the previous text and only retain the attention mechanism of the latest input or only retain the attention mechanism of part of the input, but it will lose historical context information and cannot comprehensively answer questions or generate content by integrating the full text information. In addition, there are some innovative methods that improve the position encoding method of the model and scale the position information of the rotational position encoding to the actual required length to adapt to long context situations.

[0004] In complex business scenarios, the integration of knowledge bases and prompt words requires too much manpower, time, and memory resources to pre-train the model again. Therefore, improving the context length processing capability of large models, extending the context window of the model to the required length without fine-tuning while sacrificing a small amount of model performance, and improving its application effect in power supply service command have become key requirements for current technological development.

[0005] Prior art, such as the Chinese patent with patent number "CN118427332A", discloses a method for context expansion and fine-tuning of a large food safety model based on an original text repetition mechanism. The method includes setting up a reference document, forming a group with multiple reference documents, taking the first reference document to design a question-answer pair, and giving the relevant original text in the answer to the question-answer pair; disrupting the order of the reference documents in the group, and forming a reference document list and instruction fine-tuning data; setting up a language model, importing the instruction fine-tuning data into the language model, and training and fine-tuning to obtain a long text language model. The present application has the effect of solving the problems of low question-answering accuracy and ignoring the middle part of the context information in the existing language model when facing long texts, effectively improving the question-answering accuracy of long texts, and avoiding the problem of significantly increased costs caused by manual annotation.

[0006] The problems existing in the above prior art are:

[0007] 1. Reliance on specific reference documents. If the reference documents are of low quality, incomplete, or biased, the effect and accuracy of the final model may be affected. Designing and training only based on existing reference documents may not be able to adapt to new food safety-related issues and situations that are not included in the reference documents.

[0008] 2. Although this method is designed to solve the problem of long texts, its adaptability to long texts in different fields or types may be limited. The language structure, semantic relations, etc. in different fields may be very different, resulting in poor performance of the model when applied across fields. Summary of the invention

[0009] In order to solve the above technical problems, the present invention proposes a context length extension method and system for a large power supply service command model.

[0010] The technical solution of the present invention is as follows:

[0011] The present invention proposes a context length extension method for a large power supply service command model, comprising the following steps:

[0012] Step S1: Use sine and cosine functions to add positional encoding to the input word embedding so that the large model takes word order information into account;

[0013] Step S2: In the multi-head self-attention mechanism, the encoding vector of each position is rotated, and the attention score is calculated using the position information after the rotation transformation, so as to capture the long-distance dependency by introducing the position information;

[0014] Step S3: Use the neural cut kernel feature to interpolate the position encoding after the rotation transformation, use some data to fine-tune the semantic large model, expand the context window, and reduce the loss of high-frequency information;

[0015] Step S4: construct a power supply service instruction dataset, and experts check, correct and expand the power supply service instruction dataset. Use the power supply service instruction dataset to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

[0016] As a preferred implementation, the position code is added to the input word embedding using the sine and cosine functions, and the specific formula for representing the position information is as follows:

[0017]

[0018] Among them, pos is the absolute position of the word in the sequence, m is the dimension number in the word vector, and d model is the dimension of the model; odd dimensions are calculated using the cosine function, and even dimensions are calculated using the sin function; this method allows the large model to learn the relative positions of surrounding words, because for any distance l, PE pos+l Can be represented as PE pos A linear function of .

[0019] As a preferred implementation, the attention score is calculated using the position information after the rotation transformation, and the specific process is as follows:

[0020] For an input sequence of a given length N, the corresponding embedding is represented as:

[0021]

[0022] where x i Represents the d-dimensional word embedding vector corresponding to the i-th word element;

[0023] For a vector x at position n, the encoding of its rotational position is expressed as:

[0024]

[0025] Among them, θ d / 2 is the rotation angle of vector x in dimension d;

[0026] The query vector q for each word position r and key vector k t The elements of the vector are rotated in pairs, and then the real part is taken to calculate the inner product between the query and key vectors to obtain the calculation result of self-attention. The specific calculation formula is as follows:

[0027] a(r,t)=Re[I(q,r),I(k,t)]

[0028] In the formula, Re represents the real part of the function; I(·) represents the inner product calculation function;

[0029] Expand the above formula to get:

[0030]

[0031] In the formula, j=1,2,3,…,d / 2-1.

[0032] As a preferred implementation, the interpolation of the position code after the rotation transformation by using the neural core cutting characteristic specifically includes the following steps:

[0033] Define a scaling factor s as the ratio between the extended context length L' and the original context length L:

[0034]

[0035] The rotational position encoding function of vector x is further rewritten as a function of the input vector x, position w and angle θ:

[0036]

[0037] In addition, d Defined as the wavelength of the rotated position embedding over the d-th hidden dimension:

[0038]

[0039] Where, d∈D, d=1,2,3,…,D; b is a constant; λ d represents the word length required for the d-dimensional rotation position embedding to perform a complete rotation of 2π; for a given neural network model and input data, the neural cut kernel matrix is ​​calculated to reflect the similarity between the input data and the gradient information of the model on these data, specifically for f(x, g(w), h(θ d ))Function improvements:

[0040] g(w)=w

[0041]

[0042] Introducing the parameter v on a(r,t) before calculating the softmax(·) in the self-attention mechanism has a uniform effect on the model's text understanding ability, and using the length scaling technique to further adjust the text position encoding length; the calculation of its attention weight is specifically modified as follows:

[0043]

[0044] On the other hand, the present invention also provides a context length extension system for a large power supply service command model, comprising:

[0045] The position encoding module uses sine and cosine functions to add position encoding to the input word embedding, so that the large model takes word order information into account;

[0046] The rotation transformation module rotates the encoding vector of each position in the multi-head self-attention mechanism, and uses the position information after the rotation transformation to calculate the attention score, thereby capturing long-distance dependencies by introducing position information;

[0047] The interpolation fine-tuning module uses the characteristics of the neural cut kernel to interpolate the position encoding after the rotation transformation, uses a number of data to fine-tune the semantic large model, expands the context window, and reduces the loss of high-frequency information;

[0048] The instruction fine-tuning module builds a power supply service instruction dataset. Experts check, correct and expand the power supply service instruction dataset. The power supply service instruction dataset is used to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

[0049] As a preferred implementation, the position encoding module uses sine and cosine functions to add position encoding to the input word embedding, and the specific formula for representing position information is as follows:

[0050]

[0051] Among them, pos is the absolute position of the word in the sequence, m is the dimension number in the word vector, and d model is the dimension of the model; odd dimensions are calculated using the cosine function, and even dimensions are calculated using the sin function; this method allows the large model to learn the relative positions of surrounding words, because for any distance l, PE pos+l Can be represented as PE pos A linear function of .

[0052] As a preferred implementation, the rotation transformation module uses the position information after the rotation transformation to calculate the attention score. The specific process is as follows:

[0053] For an input sequence of a given length N, the corresponding embedding is represented as:

[0054]

[0055] where x i Represents the d-dimensional word embedding vector corresponding to the i-th word element;

[0056] For a vector x at position n, the encoding of its rotational position is expressed as:

[0057]

[0058] Among them, θ d / 2is the rotation angle of vector x in dimension d;

[0059] The query vector q for each word position r and key vector k t The elements of the vector are rotated in pairs, and then the real part is taken to calculate the inner product between the query and key vectors to obtain the self-attention calculation result. The specific attention score calculation formula is as follows:

[0060] a(r,t)=Re[I(q,r),I(k,t)]

[0061] In the formula, Re represents the real part of the function; I(·) represents the inner product calculation function;

[0062] Expand the above formula to get:

[0063]

[0064] Where j = 1, 2, 3, ..., d / 2-1;

[0065] The self-attention score a(r,t) depends only on the relative position rt, where q and k are the query and key vectors for a particular attention head. At each layer, the rotational position encoding is applied to the query and key embeddings to compute the attention score.

[0066] As a preferred implementation, the interpolation fine-tuning module interpolates the position code after the rotation transformation by using the neural cut core characteristic, specifically including the following steps:

[0067] Define a scaling factor s as the ratio between the extended context length L' and the original context length L:

[0068]

[0069] The rotational position encoding function of vector x is further rewritten as a function of the input vector x, position w and angle θ:

[0070]

[0071] In addition, d Defined as the wavelength of the rotated position embedding over the d-th hidden dimension:

[0072]

[0073] Where, d∈D, d=1,2,3,…,D; b is a constant; λ drepresents the word length required for the d-dimensional rotation position embedding to perform a complete rotation of 2π; for a given neural network model and input data, the neural cut kernel matrix is ​​calculated to reflect the similarity between the input data and the gradient information of the model on these data, specifically for f(x, g(w), h(θ d ))Function improvements:

[0074] g(w)=w

[0075]

[0076] Introducing the parameter v on a(r,t) before calculating the softmax(·) in the self-attention mechanism has a uniform effect on the model's text understanding ability. The length scaling technique is used to further flexibly adjust the text position encoding length. The calculation of its attention weight is specifically modified as follows:

[0077]

[0078] On the other hand, the present invention further provides an electronic device having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for extending the context length of a large model of power supply service command as described in any embodiment of the present invention is implemented.

[0079] On the other hand, the present invention also provides a computer-readable medium for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a context length extension method for a large power supply service command model as described in any embodiment of the present invention.

[0080] The present invention has the following beneficial effects:

[0081] 1. Improve the ability to process long texts: By improving the rotational position encoding, the model can better capture long-distance dependencies and can be effectively used in the multi-head self-attention mechanism, solving the problem of limited context length in the existing technology.

[0082] 2. Improve model performance and efficiency: This invention extends the context length through interpolation method, reduces the loss of high-frequency information, and does not require additional fine-tuning of the model, thus reducing the computational burden. It can retain more conversation history information in the knowledge service process of the large model of power supply service command, improve the fluency and accuracy of the reply, and thus improve the overall intelligence level and responsiveness of the system in complex power supply service command scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0084] Figure 1 This is a schematic diagram of the method flow of Example 1;

[0085] Figure 2 Flowchart of attention score calculation for rotational position encoding;

[0086] Figure 3 Flowchart of the method for fine-tuning instructions in the field of electric power customer service. DETAILED DESCRIPTION

[0087] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0088] It should be understood that the step numbers used in this document are only for convenience of description and are not intended to limit the order in which the steps are executed.

[0089] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0090] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0091] The term "and / or" means and includes any and all possible combinations of one or more of the associated listed items.

[0092] Embodiment 1:

[0093] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following will be combined with the specific embodiments of the present application and refer to the attached Figure 1 , clearly and completely describe the technical solution of the present invention.

[0094] To solve the problems in the prior art, the present invention provides a context length extension method for a large power supply service command model, comprising the following steps:

[0095] Step S1: Use sine and cosine functions to add positional encoding to the input word embedding so that the large model takes word order information into account;

[0096] The semantic big model is based on the Transformer architecture, which is mainly divided into an embedding layer and a multi-layer decoder. It does not contain recursion or convolution, and its parallel input ignores word order information. In order to allow the model to consider the order of words in a sentence, the standard position encoding uses sine and cosine functions to add the position encoding directly to the input word embedding to represent the position information of the sequence:

[0097]

[0098] Among them, pos is the absolute position of the word in the sequence, m is the dimension number in the word vector, and d model is the dimension of the model; odd dimensions are calculated using the cosine function, and even dimensions are calculated using the sin function; this method allows the large model to learn the relative positions of surrounding words, because for any distance l, PE pos+l Can be represented as PE pos A linear function of .

[0099] Step S2: In the multi-head self-attention mechanism, the encoding vector of each position is rotated, and the attention score is calculated using the position information after the rotation transformation, so as to capture the long-distance dependency by introducing the position information;

[0100] The core idea of ​​the rotational position encoding is to rotate the encoding vector of each position to introduce position information. Specifically, for each input vector, it is regarded as a complex number and the position information is embedded in it through the rotation operation. This method can better capture long-range dependencies and can be effectively used in the multi-head self-attention mechanism.

[0101] For an input sequence of a given length N, the corresponding embedding is represented as:

[0102]

[0103] where x i Represents the d-dimensional word embedding vector corresponding to the i-th word element;

[0104] For the vector x at position n, the encoding can be expressed as:

[0105]

[0106] Among them, θd / 2 is the rotation angle of vector x in dimension d;

[0107] According to Euler's formula:

[0108] e ac =cosc+asinc

[0109] Where c represents any real number, e is the base of natural logarithm, and a is the imaginary unit in complex number; then the above exponential function can be expressed as a complex number with real part cosc ​​and imaginary part sinc;

[0110] Therefore, assuming that vector x is a two-dimensional vector, for position n, define the rotation matrix R:

[0111]

[0112] Then, for this given input sequence of length N, a rotation operation is performed before the self-attention calculation. For each even index u and odd index u+1 in the input vector x, the rotation matrix R is applied:

[0113]

[0114] This is equivalent to pairing even dimensions with odd dimensions for a two-dimensional rotation. Finally, the rotation vectors of all positions are concatenated to obtain the final rotation position encoding.

[0115] The query vector q for each word position r and key vector k t The elements of the vector are rotated in pairs, and then the real part is taken to calculate the inner product between the query and key vectors to obtain the self-attention calculation result. The specific attention score calculation formula is as follows:

[0116] a(r,t)=Re[I(q,r),I(k,t)]

[0117] In the formula, Re represents the real part of the function; I(·) represents the inner product calculation function;

[0118] Expand the above formula to get:

[0119]

[0120] Where j = 1, 2, 3, ..., d / 2-1;

[0121] The self-attention score a(r,t) depends only on the relative position rt, where q and k are the query and key vectors for a particular attention head. At each layer, the rotational position encoding is applied to the query and key embeddings to compute the attention score.

[0122] Step S3: interpolate the rotation position encoding using the neural cut kernel characteristics, use some data to fine-tune the semantic large model, expand the context window, and reduce the loss of high-frequency information;

[0123] The semantic large model is pre-trained on a fixed context length, and it is difficult for the position encoding to directly expand the context window specified during its own training. In response to the need for extended context length for power supply service command, the neural cut kernel feature is used to interpolate the rotation position encoding, and fine-tune it on a relatively small amount of data to expand the context window of the power supply service command large model. The specific steps are as follows:

[0124] First, define a scaling factor s as the ratio between the extended context length L' and the original context length L:

[0125]

[0126] Further rewrite the encoding function for vector x in step 2 as a function of input vector x, position m and rotation angle θ:

[0127]

[0128] In addition, d Defined as the wavelength of the rotated position embedding over the d-th hidden dimension:

[0129]

[0130] Where b is a constant, usually 10000. The wavelength represents the length of the word required for the d-dimensional rotation position embedding to perform a full rotation (2π). For a given neural network model and input data, the neural cut kernel matrix is ​​calculated to reflect the similarity between the input data and the gradient information of the model on these data, which is specifically expressed as f(x, g(w), h(θ d ))Function improvements:

[0131] g(w)=w

[0132]

[0133] Through the above formula, different frequency dimensions of the rotation position encoding are scaled to different degrees to achieve the purpose of reducing high frequencies and increasing low frequencies.

[0134] Introducing the parameter v on a(r,t) before calculating the softmax(·) in the self-attention mechanism has a uniform effect on the model's text understanding ability, but the length scaling technique can be used to further flexibly adjust the text position encoding length; the calculation of its attention weight is specifically modified as follows:

[0135]

[0136] By simply scaling the complex rotational position encoding by the same factor, q r and k t Constant factor Scaling effectively changes the attention mechanism without requiring additional training.

[0137] Step S4: construct a power supply service instruction dataset, and experts check, correct and expand the instruction dataset. Use the power supply service instruction dataset to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

[0138] A method for fine-tuning instructions in the field of power supply and service is designed. First, the actual knowledge service materials of the system are collected, and the data is cleaned and structured. Then the business logic of the supply and service knowledge service is constructed, and a power supply service instruction data set is formed in the form of (instruction, input, output) triples. In order to build a high-quality and diverse instruction data set, experts in the supply and service command business are invited to check and correct the wrong answers in the instruction data set, and rewrite the instruction sentences to expand the form of instructions in the data set. After the experts correct and expand the instruction data, the power supply service field instruction data set is obtained. In the model training stage, the power supply service command large model improved according to step 3 is fine-tuned on the power supply service field instruction data set, so that the large model is more adaptable to the expanded context length and can follow the instructions to perform corresponding tasks.

[0139] Embodiment 2:

[0140] This embodiment provides a context length extension system for a large power supply service command model, including:

[0141] The position encoding module uses sine and cosine functions to add position encoding to the input word embedding, so that the large model takes word order information into account;

[0142] The rotation transformation module rotates the encoding vector of each position in the multi-head self-attention mechanism and uses the position information after the rotation transformation to calculate the attention score, which can better capture long-distance dependencies.

[0143] The interpolation fine-tuning module uses the characteristics of the neural cut kernel to interpolate the rotation position encoding, uses a number of data to fine-tune the semantic large model, expands the context window, and reduces the loss of high-frequency information;

[0144] The instruction fine-tuning module builds a power supply service instruction dataset. Experts check, correct and expand the instruction dataset. The power supply service instruction dataset is used to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

[0145] Embodiment three:

[0146] This embodiment provides an electronic device having a computer program stored thereon. When the computer program is executed by a processor, a context length extension method for a large model of power supply service command as described in any embodiment of the present invention is implemented.

[0147] Embodiment 4:

[0148] This embodiment provides a computer-readable medium for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a context length extension method for a large power supply service command model as described in any embodiment of the present invention.

[0149] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0150] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0153] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A context length extension method for a large power supply service command model, characterized in that: The following steps are involved: Step S1: Use sine and cosine functions to add positional encoding to the input word embedding so that the large model takes word order information into account; Step S2: In the multi-head self-attention mechanism, the encoding vector of each position is rotated, and the attention score is calculated using the position information after the rotation transformation, so as to capture the long-distance dependency by introducing the position information; Step S3: Use the neural cut kernel feature to interpolate the position encoding after the rotation transformation, use some data to fine-tune the semantic large model, expand the context window, and reduce the loss of high-frequency information; Step S4: construct a power supply service instruction dataset, and experts check, correct and expand the power supply service instruction dataset. Use the power supply service instruction dataset to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

2. The context length extension method for a large power supply service command model according to claim 1, characterized in that: The sine and cosine functions are used to add the position code to the input word embedding. The specific formula for representing the position information is as follows: Among them, pos is the absolute position of the word in the sequence, m is the dimension number in the word vector, and d model is the dimension of the model; odd dimensions are calculated using the cosine function, and even dimensions are calculated using the sin function; this method allows the large model to learn the relative positions of surrounding words, because for any distance l, PE pos+l Can be represented as PE pos A linear function of .

3. The context length extension method for a large power supply service command model according to claim 1 is characterized in that: The attention score is calculated using the position information after rotation transformation. The specific process is as follows: For an input sequence of a given length N, the corresponding embedding is represented as: where x i Represents the d-dimensional word embedding vector corresponding to the i-th word element; For a vector x at position n, the encoding of its rotational position is expressed as: Among them, θ d / 2 is the rotation angle of vector x in dimension d; The query vector q for each word position r and key vector k t The elements of the vector are rotated in pairs, and then the real part is taken to calculate the inner product between the query and key vectors to obtain the calculation result of self-attention. The specific calculation formula is as follows: a(r, t)=Re[I(q, r), I(k, t)] In the formula, Re represents the real part of the function; I(·) represents the inner product calculation function; Expand the above formula to get: In the formula, j=1, 2, 3,…, d / 2-1.

4. The context length extension method for a large power supply service command model according to claim 1, characterized in that: The method of interpolating the position code after the rotation transformation by using the neural core cutting characteristics specifically includes the following steps: Define a scaling factor s as the ratio between the extended context length L′ and the original context length L: The rotational position encoding function of vector x is further rewritten as a function of the input vector x, position w and angle θ: In addition, d Defined as the wavelength of the rotated position embedding over the d-th hidden dimension: Where, d∈D, d=1, 2, 3, …, D; b is a constant; λ d Represents the word length required for the d-dimensional rotation position embedding to perform a full rotation of 2π; for a given neural network model and input data, the neural cut kernel matrix is ​​calculated to reflect the similarity between the input data and the gradient information of the model on these data, which is specifically manifested as an improvement on the f(x, g(w), h(θ d)) function: g(w)=w Before calculating the softmax(·) in the self-attention mechanism, the parameter v is introduced on a(r, t) to have a uniform effect on the model's text understanding ability. The length scaling technique is used to further adjust the text position encoding length. The calculation of its attention weight is specifically modified as follows:

5. A context length extension system for a large power supply service command model, characterized in that: include: The position encoding module uses sine and cosine functions to add position encoding to the input word embedding, so that the large model takes word order information into account; The rotation transformation module rotates the encoding vector of each position in the multi-head self-attention mechanism, and uses the position information after the rotation transformation to calculate the attention score, thereby capturing long-distance dependencies by introducing position information; The interpolation fine-tuning module uses the characteristics of the neural cut kernel to interpolate the position encoding after the rotation transformation, uses a number of data to fine-tune the semantic large model, expands the context window, and reduces the loss of high-frequency information; The instruction fine-tuning module builds a power supply service instruction dataset. Experts check, correct and expand the power supply service instruction dataset. The power supply service instruction dataset is used to fine-tune the instructions of the improved semantic big model so that the semantic big model can adapt to the expanded context length.

6. The context length extension system for a large power supply service command model according to claim 5 is characterized in that: The position encoding module uses sine and cosine functions to add position encoding to the input word embedding. The specific formula for position information representation is as follows: Among them, pos is the absolute position of the word in the sequence, m is the dimension number in the word vector, and d model is the dimension of the model; odd dimensions are calculated using the cosine function, and even dimensions are calculated using the sin function; this method allows the large model to learn the relative positions of surrounding words, because for any distance 1, PE pos+l Can be represented as PE pos A linear function of .

7. The context length extension system for a large power supply service command model according to claim 5 is characterized in that: The rotation transformation module uses the position information after rotation transformation to calculate the attention score. The specific process is as follows: For an input sequence of a given length N, the corresponding embedding is represented as: where x i Represents the d-dimensional word embedding vector corresponding to the i-th word element; For a vector x at position n, the encoding of its rotational position is expressed as: Among them, θ d / 2 is the rotation angle of vector x in d dimension; The query vector q for each word position r and key vector k t The elements of the vector are rotated in pairs, and then the real part is taken to calculate the inner product between the query and key vectors to obtain the self-attention calculation result. The specific attention score calculation formula is as follows: a(r, t)=Re[I(q, r), I(k, t)] In the formula, Re represents the real part of the function; I(·) represents the inner product calculation function; Expand the above formula to get: Wherein j = 1, 2, 3, ..., d / 2-1; The self-attention score a(r, t) depends only on the relative position rt, where q and k are the query and key vectors of a particular attention head. At each layer, the rotational position encoding is applied to the query and key embeddings to compute the attention score.

8. The context length extension system for a large power supply service command model according to claim 5, characterized in that: The interpolation fine-tuning module uses the neural cut core characteristics to interpolate the position code after the rotation transformation, specifically including the following steps: Define a scaling factor s as the ratio between the extended context length L′ and the original context length L: The rotational position encoding function of vector x is further rewritten as a function of the input vector x, position w and angle θ: In addition, d Defined as the wavelength of the rotated position embedding over the d-th hidden dimension: Where, d∈D, d=1, 2, 3, …, D; b is a constant; λ d Represents the word length required for the d-dimensional rotation position embedding to perform a full rotation of 2π; for a given neural network model and input data, the neural cut kernel matrix is ​​calculated to reflect the similarity between the input data and the gradient information of the model on these data, which is specifically manifested as an improvement on the f(x, g(w), h(θ d)) function: g(w)=w Introducing the parameter v on a(r,t) before calculating the softmax(·) in the self-attention mechanism has a uniform effect on the model's text understanding ability. The length scaling technique is used to further flexibly adjust the text position encoding length. The calculation of its attention weight is specifically modified as follows:

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method for extending the context length of the power supply service command large model as described in any one of claims 1 to 4 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for extending the context length of a large power supply service command model as described in any one of claims 1 to 4 is implemented.

Citation Information

Patent Citations

  • Food safety large model context extension fine tuning method based on original text retelling mechanism

    CN118427332A

  • Text natural language processing training method and system in power system operation

    CN118297131A

  • System for speech recognition text enhancement fusing multi-modal semantic invariance

    US11488586B1

  • Dam defect time-sequence image description method based on local self-attention mechanism

    WO2023217163A1