Method and system for spontaneous conversion between large language model position encoding and semantic mechanism

Abstract

The application relates to the technical field of natural language processing, and provides a spontaneous conversion method and system between position coding and semantic mechanisms of a large language model. The spontaneous conversion method between position coding and semantic mechanisms of the large language model comprises the following steps: calculating sample complexity according to the number of training samples and the input token dimension; when the sample complexity is less than a sample complexity threshold, the large language model adopts an attention mechanism based on position coding to process input data; when the sample complexity infinitely approaches the sample complexity threshold from the low sample side, the limit value of a low-rank mapping matrix of a query matrix is equal to a position coding mapping matrix; when the sample complexity infinitely approaches the sample complexity threshold from the high sample side, the limit value of the low-rank mapping matrix of the query matrix is equal to a semantic feature mapping matrix; and when the sample complexity is greater than the sample complexity threshold, the large language model adopts a semantic-enhanced attention mechanism to process input data, so that the large language model can efficiently understand and generate.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for spontaneous conversion between positional encoding and semantic mechanisms in a large language model. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, deep learning models, represented by Transformer, have achieved remarkable success in fields such as natural language processing (NLP). The core component of Transformer is the self-attention layer, which was first proposed for parallel processing of sequential data.

[0004] The self-attention layer calculates the similarity score between each pair of tokens in the sequence to achieve weighted fusion of information, thereby effectively modeling the dependencies between elements within the sequence.

[0005] In practical applications, self-attention layers typically combine two key types of information: (1) positional information: injected through positional encoding to capture the sequential features of words in a sequence; and (2) semantic information: obtained through word embedding to understand the meaning of words themselves.

[0006] In the switching process between the "positional" and "semantic" mechanisms in the self-attention layer, current technologies face two major problems. First, there is a lack of clear and reliable indicators for judging the phase transition. Existing research can only observe that changes in the training sample size trigger the switching phenomenon between the two mechanisms, but it cannot quantify and define the key nodes for the transition from the "positional mechanism" to the "semantic mechanism." This makes it difficult for the model to accurately determine when to enter the mechanism switching phase and to make timely adaptation adjustments. Second, there is no effective way to locate the switching threshold of the global optimal solution. Current technologies neither establish a correlation between sample complexity (the ratio of the number of samples to the data dimension) and mechanism switching, nor do they have a feasible path to explore the threshold by adjusting the number of samples. As a result, in practical applications, the model either fails to effectively utilize semantic information because it still relies on the positional mechanism before reaching the threshold, or it suffers performance loss because it exceeds the threshold but fails to switch to the semantic mechanism in time. It is always difficult to stably maintain the mechanism state corresponding to the global optimal solution. Summary of the Invention

[0007] To enable the attention layer to spontaneously and reasonably switch between the two paths of positional encoding and semantic similarity between word vectors, this invention provides a method and system for spontaneous conversion between positional encoding and semantic mechanisms in large language models. By using sample complexity as a phase transition indicator that large language models will clearly switch between the two attention mechanisms of position and semantics, this invention can achieve efficient understanding and generation of large language models, improving processing efficiency while ensuring the accuracy of the output data of large language models.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] The first aspect of the present invention provides a method for spontaneous conversion between positional encoding and semantic mechanisms in a large language model.

[0010] A spontaneous conversion method between positional encoding and semantic mechanisms in large language models includes:

[0011] Calculate the sample complexity based on the number of training samples and the dimension of the input token;

[0012] When the sample complexity is less than the sample complexity threshold, the large language model uses a positional encoding-based attention mechanism to process the input data; when the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the positional encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix; when the sample complexity is greater than the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data.

[0013] Furthermore, the sample complexity is expressed by the following formula:

[0014]

[0015] Where M represents the number of training samples; d represents the dimension of the input token.

[0016] Furthermore, when the sample complexity is less than the sample complexity threshold,

[0017] The low-rank mapping matrix between query matrix Q and key matrix K , Used to learn location-encoded features, i.e. , ,in Represents the positional encoding mapping matrix;

[0018] Attention weights are: ,in, It is a function that depends only on the position and distance; m and n represent particles.

[0019] Furthermore, the position-encoding-based attention mechanism is a scaled dot product attention mechanism.

[0020] Furthermore, when the sample complexity equals the sample complexity threshold, the attention weights... The determining factor smoothly transitions from positional difference to token semantic similarity, satisfying... ,in This represents the semantic similarity between n and m. This represents a semantic similarity correlation function.

[0021] Furthermore, when the sample complexity exceeds the sample complexity threshold,

[0022] The query matrix Q and the key matrix K are learned by semantic weights. , ,in, This represents the query vector for n. Represents the key vector of m; Represents the semantic features of a token; , Represents the semantic adaptation mapping matrix;

[0023] Attention weight for: .

[0024] Furthermore, the semantically enhanced attention mechanism is a multi-head attention mechanism.

[0025] Furthermore, the large language model includes the Transformer model.

[0026] A second aspect of the present invention provides a spontaneous conversion system between positional encoding and semantic mechanisms in a large language model.

[0027] A spontaneous conversion system between positional encoding and semantic mechanisms in a large language model includes:

[0028] The sample complexity calculation module is configured to calculate the sample complexity based on the number of training samples and the dimension of the input token.

[0029] The spontaneous transformation module is configured as follows: when the sample complexity is less than the sample complexity threshold, the large language model uses a positional encoding-based attention mechanism to process the input data; when the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the positional encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix; when the sample complexity is greater than the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data.

[0030] A third aspect of the present invention provides a computer device comprising:

[0031] A processor, adapted to execute computer programs;

[0032] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the spontaneous conversion method between positional encoding and semantic mechanisms of a large language model as described in the first aspect above.

[0033] Compared with the prior art, the beneficial effects of the present invention are:

[0034] This invention achieves a leap from empirical observation to theoretical analysis by establishing sample complexity as the core quantitative indicator of the phase transition in the attention mechanism of Transformer models for the first time. It leverages critical sample complexity. The precise definition clearly describes the discontinuous transition conditions from the positional mechanism to the semantic mechanism in the model, and through... and The limit formula has been mathematically verified, which solves the technical pain points of traditional technology, such as the lack of strict quantitative standards for attention mechanism transformation and the inability to theoretically trace the origin of the model's "insight moment". It provides a rigorous theoretical support framework for the internal operating logic of the attention layer of the large language model.

[0035] The critical sample complexity index proposed in this invention enables the model to proactively switch its operating mode based on the sample size, breaking through the limitation of the single and fixed attention mechanism of traditional models. When the sample size is small ( The model spontaneously activates a positional mechanism, relying on positional encoding to accurately capture sequence order information and ensure basic semantic coherence; when the sample size grows to a critical threshold ( The sample complexity index triggers a sudden change, and the model automatically transitions to a semantic mechanism, constructing accurate associations by learning the semantic weights of the query and key matrix; in the high-sample stage ( The model operates stably on a semantic mechanism, enabling in-depth mining of complex semantic relationships. This sample-driven, spontaneous transformation capability allows the model to dynamically adjust its attention focus based on the characteristics of the data input, adapting to the understanding and generation needs of different data scales without human intervention.

[0036] Based on critical sample complexity With its clear definition, this invention provides targeted guidance for model training and architecture optimization, significantly enhancing its practical application value. In the training phase, through pre-computation... The minimum sample size required to trigger the semantic mechanism can be directly determined. This approach avoids semantic learning failures due to insufficient samples and reduces computational resource waste caused by sample redundancy. Experiments have verified that it can reduce model training costs. In the architecture design phase, parameters such as the low-rank dimension r and positional encoding strength can be adjusted. The values ​​are chosen to allow the model to accurately adapt to different scenarios, such as small samples (requiring rapid activation of the semantic mechanism) and large samples (requiring guaranteed mechanism stability). Simultaneously, the transformation process of the attention mechanism can be achieved through a low-rank mapping matrix. Low-dimensional core feature matrix Real-time observation of state changes enables traceability of model operation status, improves semantic understanding accuracy and generation coherence under the same number of parameters, and achieves synergistic optimization of model performance, interpretability and training efficiency. Attached Figure Description

[0037] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0038] Figure 1 This is a flowchart illustrating the spontaneous conversion method between positional encoding and semantic mechanisms in a large language model, as shown in an embodiment of the present invention.

[0039] Figure 2 This is a structural diagram of the spontaneous conversion system between positional encoding and semantic mechanisms in a large language model, as shown in an embodiment of the present invention.

[0040] Figure 3 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] As described in the background section, the current Transformer self-attention layer suffers from two major problems in switching between "positional" and "semantic" mechanisms: (1) it lacks clear phase transition judgment indicators, making it difficult to quantify and determine the switching node; (2) it does not associate sample complexity with mechanism switching, lacking a method to locate the critical value of the optimal solution, resulting in the model being unable to switch mechanisms at the appropriate time, affecting the accuracy of the model's output results. Based on this, the present invention provides a method and system for spontaneous conversion between positional encoding and semantic mechanisms in a large language model. The present invention will be described in detail below through several embodiments.

[0045] Figure 1 This is a flowchart illustrating the spontaneous conversion method between positional encoding and semantic mechanisms in a large language model, as shown in an embodiment of the present invention; see also... Figure 1 The method includes:

[0046] Calculate the sample complexity based on the number of training samples and the dimension of the input token;

[0047] When the sample complexity is less than the sample complexity threshold, the large language model uses a positional encoding-based attention mechanism to process the input data; when the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the positional encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix; when the sample complexity is greater than the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data.

[0048] This invention provides a spontaneous conversion method between positional encoding and semantic mechanisms in large language models. It utilizes sample complexity as a phase transition indicator that large language models will clearly switch between positional and semantic attention mechanisms, enabling efficient understanding and generation of large language models. While ensuring the accuracy of the output data of large language models, it also reduces the computational load of large language models in processing input data to a certain extent, thereby improving processing efficiency.

[0049] When observing the phase transition phenomenon, it was found that when the training sample size is below a certain threshold, only positional encoding plays a role, while semantics does not. When the sample size exceeds this threshold, only semantics plays a role. Near the critical point, the model experiences fluctuations in both positional encoding and semantic mechanisms. Therefore, in a sense, this is a new phase transition between positional learning and semantic learning, which can be described using a simplified language model. Near the critical point, a critical deceleration phenomenon occurs; that is, when semantic complexity reaches a certain value, the model's understanding or generation becomes slow. At this point, the model loses resilience and is more likely to switch to other mechanisms. This embodiment uses the Transformer model.

[0050] The Transformer's self-attention layer becomes the core structure for extracting information from text data, capturing both the positional information implied by word order and understanding word semantics through word embeddings. Numerous experiments have revealed that the training process and data volume determine the specific algorithmic mechanism employed by the attention layer, but theoretical justification is lacking. This embodiment proposes using sample complexity as a clear indicator of the phase transition between the "positional" and "semantic" attention mechanisms in large language models.

[0051] The technical solution of this embodiment will be described in detail below:

[0052] 1. Self-attention layer of Transformer

[0053] An attention function can be described as a process that maps a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is calculated by a weighted sum of the values, with the weight assigned to each value calculated by a compatibility function between the query and its corresponding key.

[0054] Attention mechanisms include scaled dot product attention and multi-head attention. Among them, scaled dot product attention is a common attention mechanism. It obtains an attention score by performing a dot product operation on the query vector and the key vector, and then adjusts the attention score by a scaling factor, normalizes the score, and fuses it with the value vector in a weighted manner.

[0055] Specifically, (1) the raw attention score is calculated using the following formula:

[0056]

[0057] in, Represents the original attention score; Q represents the query vector, with dimension 1. The rectangular shape is n is the sequence length; K represents the key vector, with the same dimensions as the query vector. ), in matrix form .

[0058] (2) Introduce a scaling factor to adjust the score, expressed by the following formula:

[0059]

[0060] in, This represents the attention score after introducing a scaling factor; This represents the scaling factor in the scaling dot product attention mechanism, used to alleviate the problem of excessively large score values ​​caused by dot product operations.

[0061] (3) The normalized score is then weighted and fused with the value vector, expressed by the following formula:

[0062]

[0063] in, express The attention score after weighted fusion; V represents the value vector with dimension . The matrix form is ; This represents the normalization function that maps the attention score to... Interval.

[0064] Specifically, the multi-head attention mechanism performs multiple linear transformations on the input simultaneously. Each transformation has its own Q, K, and V matrices, and each calculates its own attention score. Finally, the outputs of all transformations are combined to obtain the final output. Therefore, the multi-head attention mechanism can learn richer and more diverse feature representations, resulting in better model performance.

[0065] (1) The linear transformation and dot product scaling of the i-th attention head are calculated using the following formula:

[0066] , ,

[0067]

[0068] in, This represents the query vector for the i-th attention head; Let i represent the key vector of the i-th attention head; This represents the value vector of the i-th attention head. , , Each represents a linear transformation matrix of the i-th attention head, satisfying: , , ,in, , , This ensures that each head dimension is evenly distributed; h represents the number of attention heads; This represents the output of the i-th attention head; express The attention score after weighted fusion; This represents the scaling factor in the multi-head attention mechanism.

[0069] (2) Concatenate all attention head outputs, expressed by the following formula:

[0070]

[0071] in, This represents the fusion result of all attention heads; This indicates a splicing operation.

[0072] (3) The final output is obtained by linear transformation, expressed by the following formula:

[0073]

[0074] in, Indicates the final output; This represents the final output linear transformation matrix, with dimension 1. ; This is the final output dimension for multi-head attention.

[0075] In some embodiments, the word embedding and position encoding vectors are fused and projected into a query vector. Although they have the same vocabulary, different positions will produce different attention scores. Position encoding tells the attention mechanism the relationship between each pair of positions, so that the Transformer model understands both semantics and position.

[0076] Specifically, (1) the input representation of the word embedding vector is expressed by the following formula:

[0077]

[0078] in, The input representation of the word embedding vector; Represents word embedding vectors.

[0079] (2) The location code is calculated using the following formula:

[0080]

[0081] in, The input representation of the position-encoded vector; This represents the positional encoding vector.

[0082] (3) The fusion calculation of word embedding and positional encoding is performed using the following formula:

[0083]

[0084] in, Represents the fused feature vector; This indicates an element-wise addition operation.

[0085] (4) Project the query vector, which is calculated using the following formula:

[0086]

[0087] in, Represents the query vector; This represents a weight matrix.

[0088] (5) Calculate the attention score using the following formula:

[0089]

[0090] Where K represents the key vector; This represents the scaling factor.

[0091] 2. Location-Semantic Representation Phase Transition Technology

[0092] The self-attention layer of a Transformer can be viewed as a continuous-time interactive particle system. In this system, each token in the input sequence is considered a particle. Here, t represents time, which can be understood as the depth or number of layers in the network. These particles interact and evolve over time through a self-attention mechanism. Under the influence of the self-attention mechanism, particles representing tokens tend to cluster onto specific extreme objects over time (i.e., as the network layers deepen). The positions of these clusters are determined by the initial tokens, and the representations learned by the Transformer are context-aware.

[0093] The specific process is as follows: The self-attention layer of the Transformer model is mapped to a continuous-time interactive particle system, defined as follows:

[0094] (1) Each token in the input sequence corresponds to a particle in the particle system, denoted as ,in, N is the length of the input sequence; This represents the time variable and corresponds to the network depth (number of layers) of the Transformer model.

[0095] (2) The interaction between particles is defined by the self-attention mechanism. The evolution of the particle state with time t represents the iterative update process of the token features in each layer of the Transformer model, and is expressed by the following formula:

[0096]

[0097] in, The expression represents the interaction strength between particle n and particle m at time t, corresponding to the self-attention weight; the right side of the equation represents the tendency of a particle to evolve towards the system's average state under the influence of its neighboring particles.

[0098] With the input token dimension d increasing proportionally to the number of training samples M, and the sample complexity... In the high-dimensional limit where constants are maintained, an analytically analyzable low-rank dot product attention model is constructed, exhibiting a phase transition phenomenon under the high-dimensional limit. This embodiment proposes using sample complexity as an indicator of a clear phase transition between the "positional" and "semantic" attention mechanisms. By adjusting the number of samples, the model discovers that when the sample complexity (the ratio of the number of samples to the data dimension) exceeds a critical value, the global optimal solution switches from a positional calculation mechanism to a semantic similarity enhancement mechanism between word vectors. The model's process is as follows: (a) In the low-sample stage: the model mainly relies on positional encoding, only capturing the sequential information of words, resulting in limited generalization ability. (b) At the phase transition critical point: as the ratio of the training sample size to the data dimension changes, the model's learning mechanism undergoes a sudden leap. There is a clear critical sample complexity; exceeding this threshold, the model undergoes a sudden shift from positional to semantic, similar to a phase transition phenomenon in physics. (c) In the high-sample stage: the model learns the semantic weights of the query and key matrix to achieve semantic-based attention, enabling a more accurate understanding of semantic relationships and significantly improving performance.

[0099] The specific process is as follows:

[0100] (1) Construction of low-rank dot product attention model

[0101] In the high-dimensional extreme scenario, where the input token dimension d increases proportionally to the number of training samples M, and the sample complexity increases... By keeping constants, we construct an analytical low-rank dot product attention model.

[0102] 1) Define the query matrix Key matrix And both Q and K satisfy the low-rank assumption, i.e. , ,in, , , represents the core matrix that realizes low-rank decomposition; Indicates a low-rank dimension; , This represents a low-rank mapping matrix.

[0103] 2) Dot product attention weights The following formula is used for calculation:

[0104]

[0105] in, , Let n be the query vector of particle n at time t and m be the key vector of particle m. This represents the scaling factor.

[0106] (2) Definition of sample complexity

[0107] The sample complexity c is defined as the ratio of the number of training samples M to the dimension d of the input tokens, i.e.:

[0108]

[0109] Sample complexity c serves as a quantitative indicator of the phase transition between the "positional attention mechanism" and the "semantic attention mechanism" in the Transformer model.

[0110] In this embodiment, the attention mechanism of the Transformer model exhibits a clear phase transition phenomenon as the sample complexity c changes. The specific evolution process is divided into: low sample stage, phase transition critical point, and high sample stage.

[0111] (1) Low sample stage ( , (for critical sample complexity)

[0112] When the sample complexity is below the critical value At this time, the model's attention mechanism mainly relies on positional encoding, with the following specific features:

[0113] 1) The low-rank mapping matrix between query matrix Q and key matrix K , The main learning focus is on position-encoded features, i.e. , ,in, This represents the positional encoding mapping matrix.

[0114] 2) Attention weight It is mainly determined by the position difference of the tokens, satisfying ,in, It is a function that is only related to location distance.

[0115] 3) At this stage, the model can only capture the order information of words, has a weak ability to model semantic relationships, and has limited generalization performance.

[0116] Based on the critical value of sample complexity By defining the quantitative aspects, this invention can directly provide targeted guidance for the training strategies and structural design of large language models: during the training phase, pre-computation can be used... Determine the minimum sample size to trigger an "insight moment". To avoid semantic learning failures due to insufficient training or wasted computational resources due to sample redundancy, and to reduce model training costs; during the structural design phase, the low-rank dimension r or the positional encoding strength can be adjusted. The value of r can be adjusted to suit different data sizes and task requirements (e.g., in small sample scenarios, increasing r can reduce the impact of data size). This accelerates the initiation of semantic learning. At the same time, based on the interpretability of the phase transition mechanism, the attention layer structure can be optimized in a targeted manner to enhance the semantic feature extraction capability and improve the accuracy of semantic understanding of the model under the same number of parameters. This provides a clear technical path for the development of high-performance, highly interpretable large language models.

[0117] (2) Phase transition critical point ( )

[0118] As the number of training samples increases, the sample complexity c reaches a critical value. At this time, the model learning mechanism undergoes a sudden transition, with the following specific characteristics:

[0119] 1) Low-rank mapping matrix , The learning objective shifts abruptly from position-encoded features to semantic features, and there exists That is, when the sample complexity approaches infinitely close from the low sample side When querying the low-rank mapping matrix of the matrix. The limit value is equal to the position encoding mapping matrix. ; That is, when the sample complexity approaches infinitely from the high sample side When querying the low-rank mapping matrix of the matrix. The limit value is equal to the semantic feature mapping matrix. .

[0120] 2) Attention weight The determining factor smoothly transitions from positional difference to token semantic similarity, satisfying... ,in This represents the semantic similarity between n and m. This represents a semantic similarity correlation function.

[0121] 3) This stage corresponds to the phase transition phenomenon in physics, and the model reaches a critical sample complexity. This achieves a qualitative change in the attention mechanism.

[0122] This invention, by constructing a continuous-time interactive particle system and a low-rank dot product attention model under the high-dimensional limit, for the first time mathematically quantifies and defines the essence of the "insight moment" in large language models—that is, when the sample complexity reaches a critical value. At that time, the model's attention mechanism undergoes a discontinuous phase transition from "position-driven" to "semantic-driven." (This is achieved through...) and The extreme characterization clearly reveals the intrinsic triggering conditions and state transition laws of "insight", solving the technical pain point of the lack of quantifiable and analytical theoretical support for "insight moments" in traditional technologies, and providing a rigorous theoretical explanation framework for the abrupt change phenomenon of semantic understanding ability of large language models.

[0123] (3) High sample stage ( )

[0124] When the sample complexity exceeds the critical value At this time, the model's attention mechanism completely switches to a semantic-based learning mode, specifically manifested as follows:

[0125] 1) The query matrix Q and the key matrix K are learned by semantic weights to achieve... , ,in, Represents the semantic features of a token; , This represents the semantic adaptation mapping matrix.

[0126] 2) Attention weight Dominated by semantic similarity between tokens, that is:

[0127]

[0128] It can accurately capture complex semantic relationships such as synonyms and hyponyms.

[0129] 3) The model performance is significantly improved in this stage, with strong context awareness and semantic understanding capabilities.

[0130] This invention clarifies the intrinsic transformation path of the model from shallow location learning to deep semantic understanding: in the low-sample stage ( Sequence information is captured by relying on position encoding, at the phase transition critical point ( ) completes a sudden leap in the learning mechanism, in the high-sample stage ( Attention associations are constructed based on semantic similarity. This is achieved by tracking the low-rank mapping matrix. Limit state changes and low-dimensional core feature matrix The evolutionary trajectory of the model can be directly traced to the transformation process of the model's learning mechanism, transforming the originally black-box semantic understanding formation process into an observable and verifiable quantitative indicator. This significantly improves the interpretability of the core learning mechanism of the large language model and breaks through the technical limitations of the traditional model's semantic learning process being untraceable and difficult to interpret.

[0131] The above combination Figure 1 The spontaneous conversion method between positional encoding and semantic mechanism of large language model provided in the embodiments of the present invention has been described in detail. Next, the spontaneous conversion system between positional encoding and semantic mechanism of large language model provided in the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0132] Figure 2 This is a schematic diagram illustrating the structure of a spontaneous conversion system between positional encoding and semantic mechanisms in a large language model, as shown in an embodiment of the present invention. (Refer to...) Figure 2 The system described in this invention includes:

[0133] The sample complexity calculation module is configured to calculate the sample complexity based on the number of training samples and the dimension of the input token.

[0134] The spontaneous transformation module is configured as follows: when the sample complexity is less than the sample complexity threshold, the large language model uses a positional encoding-based attention mechanism to process the input data; when the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the positional encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix; when the sample complexity is greater than the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data.

[0135] In some embodiments, the sample complexity is expressed by the following formula:

[0136]

[0137] Where M represents the number of training samples; d represents the dimension of the input token.

[0138] In some embodiments, when the sample complexity is less than a sample complexity threshold,

[0139] The low-rank mapping matrix between query matrix Q and key matrix K , Used to learn location-encoded features, i.e. , ,in Represents the positional encoding mapping matrix;

[0140] Attention weights are: ,in, It is a function that depends only on the position and distance; m and n represent particles.

[0141] In some embodiments, the position-encoded attention mechanism is a scaled dot product attention mechanism.

[0142] In some embodiments, attention weights are applied when the sample complexity equals a sample complexity threshold. The determining factor smoothly transitions from positional difference to token semantic similarity, satisfying... ,in This represents the semantic similarity between n and m. This represents a semantic similarity correlation function.

[0143] In some embodiments, when the sample complexity is greater than a sample complexity threshold,

[0144] The query matrix Q and the key matrix K are learned by semantic weights. , ,in, This represents the query vector for n. Represents the key vector of m; Represents the semantic features of a token; , Represents the semantic adaptation mapping matrix;

[0145] Attention weight for: .

[0146] In some embodiments, the semantically enhanced attention mechanism is a multi-head attention mechanism.

[0147] In some embodiments, the large language model includes a Transformer model.

[0148] According to embodiments of the present invention, the spontaneous conversion system between positional encoding and semantic mechanisms of a large language model can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the spontaneous conversion system between positional encoding and semantic mechanisms of a large language model are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0149] See Figure 3The diagram shows the structure of a computer device, which includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium are connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the computer device's memory. The computer-readable storage medium stores computer programs, including program instructions, and the processor executes the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve the corresponding steps in the embodiment of the spontaneous conversion method between large language model positional encoding and semantic mechanisms.

[0150] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0151] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0155] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for spontaneous conversion between positional encoding and semantic mechanisms in a large language model, characterized in that, include: Calculate the sample complexity based on the number of training samples and the dimension of the input token; When the sample complexity is less than the sample complexity threshold, the large language model uses a position-encoding-based attention mechanism to process the input data. When the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the position encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix. When the sample complexity exceeds the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data. The sample complexity is expressed by the following formula: Where M represents the number of training samples; d represents the dimension of the input token; When the sample complexity is less than the sample complexity threshold, the low-rank mapping matrix between the query matrix Q and the key matrix K is... , Used to learn location-encoded features, i.e. , ,in This represents the positional encoding mapping matrix; the attention weights are: ,in, It is a function that depends only on position and distance; m and n represent particles; When the sample complexity equals the sample complexity threshold, the attention weights... The determining factor smoothly transitions from positional difference to token semantic similarity, satisfying... ,in This represents the semantic similarity between n and m. This represents a semantic similarity correlation function; When the sample complexity exceeds a threshold, the query matrix Q and the key matrix K learn semantic weights to achieve... , ,in, This represents the query vector for n. Represents the key vector of m; Represents the semantic features of a token; , Semantic adaptation mapping matrix; attention weights for: .

2. The spontaneous conversion method between positional encoding and semantic mechanisms in a large language model according to claim 1, characterized in that, The position-encoding-based attention mechanism is a scaled dot product attention mechanism.

3. The spontaneous conversion method between positional encoding and semantic mechanisms in a large language model according to claim 1, characterized in that, The semantic enhancement attention mechanism is a multi-head attention mechanism.

4. The spontaneous conversion method between positional encoding and semantic mechanisms in a large language model according to claim 1, characterized in that, The large language model includes the Transformer model.

5. A spontaneous conversion system between positional encoding and semantic mechanisms in a large language model, characterized in that, include: The sample complexity calculation module is configured to calculate the sample complexity based on the number of training samples and the dimension of the input token. The spontaneous transformation module is configured as follows: when the sample complexity is less than the sample complexity threshold, the large language model uses a positional encoding-based attention mechanism to process the input data; when the sample complexity approaches the sample complexity threshold infinitely from the low sample side, the limit value of the low-rank mapping matrix of the query matrix is ​​equal to the positional encoding mapping matrix; when the sample complexity approaches the sample complexity threshold infinitely from the high sample side, the limit value of the low-rank mapping matrix of the query matrix is ​​equal to the semantic feature mapping matrix. When the sample complexity exceeds the sample complexity threshold, the large language model uses a semantically enhanced attention mechanism to process the input data. The sample complexity is expressed by the following formula: Where M represents the number of training samples; d represents the dimension of the input token; When the sample complexity is less than the sample complexity threshold, the low-rank mapping matrix between the query matrix Q and the key matrix K is... , Used to learn location-encoded features, i.e. , ,in This represents the positional encoding mapping matrix; the attention weights are: ,in, It is a function that depends only on position and distance; m and n represent particles; When the sample complexity equals the sample complexity threshold, the attention weights... The determining factor smoothly transitions from positional difference to token semantic similarity, satisfying... ,in This represents the semantic similarity between n and m. This represents a semantic similarity correlation function; When the sample complexity exceeds a threshold, the query matrix Q and the key matrix K learn semantic weights to achieve... , ,in, This represents the query vector for n. Represents the key vector of m; Represents the semantic features of a token; , Semantic adaptation mapping matrix; attention weights for: .

6. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the spontaneous conversion method between positional encoding and semantic mechanisms of a large language model as described in any one of claims 1-4.

Images