Method and device for generating mask matrix of large language model

Abstract

The application relates to the technical field of artificial intelligence, in particular to a large language model mask matrix generation method and device, wherein the method comprises the following steps: based on an input word vector matrix of a Transformer architecture, a corresponding semantic correlation score matrix is calculated; the standard deviation of each element in the semantic correlation score matrix is calculated; if the standard deviation is less than a preset standard deviation threshold, the semantic irrelevant position and the semantic relevant position are determined based on a target sparsity, otherwise, the semantic irrelevant position and the semantic relevant position are determined based on a target threshold; and a mask matrix is determined. Thus, the problems in the related art that when a long text is input, the redundancy is high, the invalid full-attention calculation is too much, the reasoning efficiency is low, the noise interference is serious, the fixed mask disperses the attention, dilutes the core semantics, affects the generation understanding, and lacks content perception, the mask is fixed and cannot be dynamically adjusted, the calculation overhead and semantic demand are not matched, and the performance is difficult to optimize are solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method and apparatus for generating a large language model mask matrix. Background Technology

[0002] Current mainstream generative large language models generally adopt the Transformer architecture as the core structure. The self-attention mechanism is the core component of this architecture. It achieves efficient extraction of contextual semantic information by calculating the degree of semantic association between words in the input sequence. The masking mechanism is a key technology to constrain the scope of self-attention calculation and ensure the compliance of text generation logic.

[0003] In related technologies, the Transformer architecture generally adopts a static masking mechanism, which is divided into two categories: one is causal masking, which uses a fixed-format lower triangular matrix structure to force each input word to only pay attention to itself and words in the preceding text, and prohibits it from paying attention to words in the future, thus adapting to the logical requirements of unidirectional text generation; the other is fixed sparse masking, such as sliding window masking, which only allows the current word to pay attention to neighboring words within a fixed distance range, thereby reducing some of the computational load by limiting the attention range.

[0004] However, in related technologies, when inputting long texts, computational redundancy is high, there are many invalid full attention calculations, and inference efficiency is low; noise interference is severe, fixed masks distract attention, dilute core semantics, and affect generation and understanding; and there is a lack of content awareness, with fixed masks that cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making it difficult to optimize performance, and improvements are urgently needed. Summary of the Invention

[0005] This application provides a method and apparatus for generating a large language model mask matrix to solve the following problems in related technologies: high computational redundancy, many invalid full attention calculations, and low inference efficiency when inputting long texts; severe noise interference, fixed masks distract attention, dilute core semantics, and affect generation and understanding; and lack of content awareness, fixed masks cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making it difficult to optimize performance.

[0006] The first aspect of this application provides a method for generating a mask matrix for a large language model, wherein the large language model is based on the Transformer architecture. The method includes the following steps: calculating a semantic relevance score matrix based on the input word vector matrix of the Transformer architecture; calculating the standard deviation of each element in the semantic relevance score matrix; if the standard deviation is less than a preset standard deviation threshold, determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on a target sparsity rate; otherwise, determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold; and determining a mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions.

[0007] Optionally, in one embodiment of this application, the step of calculating the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture includes: determining the corresponding query vector and key vector based on the input word vector matrix; determining the corresponding dimensionality reduction matrix based on the query vector and the key vector; calculating the corresponding dimensionality reduction query vector and dimensionality reduction key vector based on the query vector, the key vector, and the dimensionality reduction matrix; and calculating the semantic relevance score matrix based on the dimensionality reduction query vector and the dimensionality reduction key vector.

[0008] Optionally, in one embodiment of this application, determining the mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions includes: modifying the elements corresponding to the semantically irrelevant positions in the semantic relevance score matrix to a first target value, and modifying the elements corresponding to the semantically relevant positions in the semantic relevance score matrix to a second target value; and determining the mask matrix based on the first target value, the second target value, the semantically irrelevant positions, and the semantically relevant positions.

[0009] Optionally, in one embodiment of this application, the method further includes: obtaining a causal mask matrix adapted to the self-attention mechanism of the Transformer decoder; determining the superposition method of the causal mask matrix and the mask matrix; and determining a composite mask matrix corresponding to the input word vector matrix based on the causal mask matrix, the mask matrix, and the superposition method.

[0010] Optionally, in one embodiment of this application, before determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold, the method further includes: determining an initial threshold corresponding to the target threshold based on the input word vector matrix; obtaining the training parameters and training results corresponding to the current training round of the large language model; and correcting the initial threshold based on the training parameters and the training results to obtain the corrected target threshold.

[0011] A second aspect of this application provides an apparatus for generating a mask matrix for a large language model, the large language model being based on a Transformer architecture. The apparatus includes: a first calculation module for calculating a corresponding semantic relevance score matrix based on an input word vector matrix from the Transformer architecture; a second calculation module for calculating the standard deviation of each element in the semantic relevance score matrix; a first determination module for determining semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on a target sparsity rate when the standard deviation is less than a preset standard deviation threshold; otherwise, determining semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold; and a second determination module for determining a mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions.

[0012] Optionally, in one embodiment of this application, the first calculation module includes: a first determining unit, configured to determine a corresponding query vector and key vector based on the input word vector matrix; a second determining unit, configured to determine a corresponding dimensionality reduction matrix based on the query vector and the key vector; a first calculation unit, configured to calculate a corresponding dimensionality reduction query vector and dimensionality reduction key vector based on the query vector, the key vector, and the dimensionality reduction matrix; and a second calculation unit, configured to calculate the semantic relevance score matrix based on the dimensionality reduction query vector and the dimensionality reduction key vector.

[0013] Optionally, in one embodiment of this application, the second determining module includes: a modification unit, configured to modify the elements corresponding to semantically irrelevant positions in the semantic relevance score matrix to a first target value, and modify the elements corresponding to semantically relevant positions in the semantic relevance score matrix to a second target value; and a third determining unit, configured to determine the mask matrix based on the first target value, the second target value, the semantically irrelevant positions, and the semantically relevant positions.

[0014] Optionally, in one embodiment of this application, it further includes: an acquisition module, used to acquire a causal mask matrix adapted to the self-attention mechanism of the Transformer decoder; a third determination module, used to determine the superposition method of the causal mask matrix and the mask matrix; and a fourth determination module, used to determine the composite mask matrix corresponding to the input word vector matrix based on the causal mask matrix, the mask matrix, and the superposition method.

[0015] Optionally, in one embodiment of this application, it further includes: a fifth determining module, used to determine an initial threshold corresponding to the target threshold based on the input word vector matrix before determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold; and a correction module, used to obtain the training parameters and training results corresponding to the current training round of the large language model, and correct the initial threshold based on the training parameters and the training results to obtain the corrected target threshold.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for generating a large language model mask matrix as described in the above embodiments.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating a large language model mask matrix.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, implements the above-described method for generating a large language model mask matrix.

[0019] This application's embodiments can calculate the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture, and then calculate the corresponding standard deviation. If the standard deviation is less than a preset standard deviation threshold, semantically irrelevant and semantically relevant positions are determined based on the target sparsity. Otherwise, semantically irrelevant and semantically relevant positions are determined based on the target threshold, and then the corresponding mask matrix is ​​determined. This can adapt to different score distribution scenarios, accurately distinguish between effective and redundant semantics, improve the rationality and applicability of the mask, and reduce the computational overhead and noise interference caused by irrelevant semantics. Therefore, it solves the problems in related technologies, such as high computational redundancy, many invalid full attention calculations, and low inference efficiency when inputting long texts; severe noise interference, fixed masks distracting attention, diluting core semantics, and affecting generation and understanding; and lack of content awareness, fixed masks that cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making performance optimization difficult.

[0020] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0021] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a method for generating a large language model mask matrix according to an embodiment of this application; Figure 2 This is a schematic diagram comparing the effects of a dynamic mask and a traditional mask according to an embodiment of this application; Figure 3 A flowchart illustrating the working principle of a method for generating a large language model mask matrix according to an embodiment of this application; Figure 4 This is a block diagram illustrating the overall architecture of a method for generating a large language model mask matrix according to an embodiment of this application. Figure 5 This is a block diagram of a device for generating a large language model mask matrix according to an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0022] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0023] Before introducing the method for generating the large language model mask matrix proposed in the embodiments of this application, the relevant terms involved in the embodiments of this application will be explained.

[0024] Transformer is a deep learning model architecture based on self-attention mechanism. It achieves sequential data modeling through self-attention mechanism and has advantages such as strong parallel computing capability and good long-distance dependency modeling effect.

[0025] The Transformer employs an encoder-decoder architecture. The encoder converts the input sequence into a high-dimensional feature representation containing contextual information. The decoder generates output tokens one by one in an autoregressive manner based on the encoded features and the generated sequence. It can consist of input embedding, positional encoding, self-attention mechanisms, multi-head self-attention, feedforward neural networks, residual connections and layer normalization, masked multi-head self-attention, and encoder-decoder attention; this application does not impose specific limitations.

[0026] Input embedding can be understood as mapping discrete words in a text sequence to continuous dense vectors, so that text information can be numerically processed by the model.

[0027] Positional encoding injects positional information into a sequence through methods such as trigonometric functions, enabling the model to perceive the order of words in the sequence.

[0028] The self-attention mechanism calculates the correlation between query vectors, key vectors, and value vectors, enabling each lexical unit to aggregate global contextual information. It is the core mechanism of Transformer.

[0029] Multi-head self-attention executes self-attention in parallel across multiple subspaces, enabling the model to learn features from different dimensions such as syntax, semantics, and referential relationships simultaneously, thereby improving feature representation capabilities.

[0030] Feedforward neural networks perform independent nonlinear transformations on the features of each word, further enhancing feature representation.

[0031] Residual connections and layer normalization alleviate the gradient vanishing problem in deep networks by using residual connections, and layer normalization stabilizes the training process and accelerates model convergence.

[0032] Masked multi-head self-attention is mainly applied to decoder structures. By introducing causal masks, the current word can only focus on itself and words in its historical position, and cannot access information about future positions, thus ensuring the autoregressivity of the generation process.

[0033] The encoder-decoder attention setting is in the decoder, whose query vector comes from the decoder and key and value vectors come from the encoder, allowing the generation process to dynamically focus on key information in the input sequence.

[0034] The following describes a method and apparatus for generating a large language model mask matrix according to embodiments of this application, with reference to the accompanying drawings. Addressing the issues mentioned in the background art, such as high computational redundancy, numerous invalid full attention calculations, low inference efficiency, severe noise interference, and fixed masks that distract attention, dilute core semantics, and affect generation and understanding when inputting long texts, as well as the lack of content awareness, fixed masks that cannot be dynamically adjusted, leading to a mismatch between computational overhead and semantic requirements, and difficulty in performance optimization, this application provides a method for generating a large language model mask matrix. In this method, the corresponding semantic relevance score matrix can be calculated based on the input word vector matrix of the Transformer architecture, and then the corresponding standard deviation can be calculated. If the standard deviation is less than a preset standard deviation threshold, semantically irrelevant and semantically relevant positions are determined based on a target sparsity rate; otherwise, semantically irrelevant and semantically relevant positions are determined based on a target threshold, thereby determining the corresponding mask matrix. This method can adapt to different score distribution scenarios, accurately distinguish between effective and redundant semantics, improve the rationality and applicability of the mask, and reduce computational overhead and noise interference caused by irrelevant semantics. This solves the problems in related technologies, such as high computational redundancy, excessive invalid full attention computation, low inference efficiency, severe noise interference, fixed mask distracting attention, diluting core semantics, and affecting generation and understanding; and lack of content awareness, fixed mask that cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making it difficult to optimize performance.

[0035] Specifically, Figure 1 This is a flowchart of a method for generating a large language model mask matrix according to an embodiment of this application.

[0036] like Figure 1 As shown, the method for generating the mask matrix of this large language model, which is based on the Transformer architecture, includes the following steps: In step S101, the corresponding semantic relevance score matrix is ​​calculated based on the input word vector matrix of the Transformer architecture.

[0037] It is understood that in the embodiments of this application, the Transformer architecture is a core branch of the Transformer network structure and is a dedicated infrastructure for large language models (such as the GPT series, LLaMA series, etc., which are not specifically limited in this application). It is different from the encoder-decoder architecture and the pure encoder architecture. This architecture allows the word at the current position to pay attention to itself and the words at the previous position through built-in causal mask constraints.

[0038] The input word vector matrix can be understood as a digital vector matrix obtained by transforming the input text sequence of a large language model through word embedding layer after word segmentation. The rows of the matrix correspond to individual words / segments in the input sequence, and the columns of the matrix correspond to the feature dimensions of the word vectors. Overall, it is used to convert the input data in text form into numerical data that the model can compute, and it is the direct input data for the self-attention mechanism.

[0039] The semantic relevance score matrix is ​​a square matrix, with the number of rows and columns equal to the length of the input text sequence. Any position within the matrix... The corresponding element value represents the first element in the sequence. The word and the first The semantic relevance score between words is used as the core basis for subsequent filtering of semantically irrelevant and semantically relevant positions.

[0040] As one possible implementation, embodiments of this application can calculate the semantic similarity between every two words based on the input word vector matrix of the Transformer architecture, and then output a corresponding semantic relevance score matrix.

[0041] For example, the Transformer architecture in this application embodiment receives a preprocessed input word vector matrix, which serves as the initial input for multi-head attention and has a fixed sequence length and feature dimension. Furthermore, the embodiments of this application can generate a corresponding semantic relevance score matrix by vector dot product based on the input word vector matrix. This can accurately quantify the semantic relationship between any two words in the input word sequence, providing reliable data support for subsequent selection of semantically related positions based on standard deviation, target sparsity, or target threshold. At the same time, it is fully compatible with the unidirectional attention constraint logic of the Transformer architecture, ensuring the compatibility of the subsequent mask matrix with the model architecture.

[0042] Optionally, in one embodiment of this application, calculating the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture includes: determining the corresponding query vector and key vector based on the input word vector matrix; determining the corresponding dimensionality reduction matrix based on the query vector and key vector; calculating the corresponding dimensionality reduction query vector and dimensionality reduction key vector based on the query vector, key vector, and dimensionality reduction matrix; and calculating the semantic relevance score matrix based on the dimensionality reduction query vector and dimensionality reduction key vector.

[0043] It is understood that, in this embodiment of the application, the query vector can be understood as a matrix generated by linear mapping based on the input word vector matrix, used to represent the attention demand of the current word element. The formula for calculating the query vector can be, but is not limited to, the following: , in, Represents the query vector. This represents the input word vector matrix. This represents a trainable weight matrix.

[0044] A key vector can be understood as a matrix generated through a linear mapping based on the input word vector matrix, used to store the feature labels of all words in the sequence. The formula for calculating the key vector can be, but is not limited to, the following: , in, Represents the key vector. This represents a trainable weight matrix.

[0045] A dimensionality reduction matrix can be understood as a matrix that can reduce the dimensionality of query vectors and key vectors. The specific settings can be configured by those skilled in the art according to the actual situation, and this application does not impose any specific restrictions.

[0046] In some embodiments, this application first generates corresponding query vectors and key vectors based on the input word vector matrix. Then, based on the dimension information of the query vectors and key vectors, a dimensionality reduction matrix is ​​determined to reduce the vector dimension. The dimensionality reduction matrix is ​​then used to project the query vectors and key vectors into the dimensionality-reduced matrix to obtain the dimensionality-reduced query vectors and key vectors. Subsequently, similarity is calculated based on the dimensionality-reduced query vectors and key vectors to obtain the corresponding semantic relevance score matrix for subsequent attention weighting operations.

[0047] For example, in the embodiments of this application, the corresponding query vector can be determined first based on the input word vector matrix. and key vector Then, a dimension reduction projection is performed using a dimension reduction matrix to generate the corresponding dimension reduction query vector. and dimensionality reduction key vector Estimate semantic space at extremely low cost and through computation and The vector dot product generates the semantic relevance score matrix. .

[0048] In step S102, the standard deviation of each element in the semantic relevance score matrix is ​​calculated.

[0049] In some embodiments, the present application can calculate the standard deviation of each element in the semantic relevance score matrix. The main steps include: extracting all score elements from the semantic relevance score matrix to form a one-dimensional score sequence; calculating the mean and variance of the sequence based on the one-dimensional score sequence; and then calculating the standard deviation based on the variance to obtain a standard deviation value that characterizes the dispersion of the semantic relevance scores.

[0050] For example, in this embodiment of the application, the semantic relevance score matrix is ​​set as follows: , to all in the matrix Each element expands into a score sequence. (in, Then calculate the standard deviation of all elements. Its expression can be, but is not limited to, as: , in, This is the mean of all semantic relevance scores.

[0051] Optionally, in one embodiment of this application, before determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold, the method further includes: determining the initial threshold corresponding to the target threshold based on the input word vector matrix; obtaining the training parameters and training results corresponding to the current training round of the large language model; and correcting the initial threshold based on the training parameters and training results to obtain the corrected target threshold.

[0052] It is understood that, in the embodiments of this application, the initial threshold can be understood as a basic threshold determined based on the inherent properties of the input word vector matrix, which is the initial benchmark value of the target threshold. There is no intervention from the dynamic information of model training, and it only adapts to the static features of the input data itself, serving as the initial basis for subsequent threshold correction.

[0053] Training parameters may include, but are not limited to, the number of model attention heads, word vector feature dimensions, learning rate, weight decay coefficient, upper limit of sequence length, and preset sparsity parameters, etc. This application does not impose specific restrictions.

[0054] The training results can be understood as the semantic recognition accuracy, attention weight distribution convergence, computational redundancy rate, loss function value, semantic relevance discrimination accuracy, etc., obtained by the large language model after completing the current round of training. This application does not impose specific restrictions and is used to intuitively reflect the adaptation effect of the current initial threshold, which is the core feedback basis for threshold correction.

[0055] In some embodiments, before using the target threshold to determine which positions in the semantic relevance score matrix belong to semantically irrelevant positions and which belong to semantically relevant positions, the present application embodiments can preliminarily determine the initial threshold corresponding to the target threshold based on the input word vector matrix, obtain the training parameters and training results corresponding to the current training round of the large language model, and then correct the initial threshold based on the obtained training parameters and training results to obtain the corrected target threshold.

[0056] For example, in order to overcome the poor adaptability of fixed thresholds and ensure that the target threshold fits the input data features and the real-time training state of the model, and to ensure the accuracy of filtering semantically related and irrelevant positions, this application embodiment can first rely on the input word vector matrix, such as by extracting the sequence length, feature dimension, vector value distribution variance, and sequence word density of the input word vector matrix, to calculate an initial threshold that fits the current input data. Then, the training parameters of the current training round of the large language model are retrieved in real time, and the training results of the current round are obtained synchronously. Based on the training parameters, the current training stage of the model is determined, and the initial threshold is corrected in combination with the training results to obtain the corrected target threshold. This ensures that the target threshold matches both the static features of the input word vector matrix and the real-time training state of the model, avoiding the problem of insufficient filtering accuracy caused by fixed thresholds.

[0057] Furthermore, the embodiments of this application can accurately divide the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix according to the target threshold, providing accurate positional basis for subsequent mask matrix generation, thereby realizing adaptive optimization of the mask matrix, reducing the invalid computation of the self-attention mechanism, and improving the overall inference efficiency and semantic understanding accuracy of the model.

[0058] In step S103, if the standard deviation is less than the preset standard deviation threshold, the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix are determined based on the target sparsity rate; otherwise, the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix are determined based on the target threshold.

[0059] In some embodiments, this application can determine whether the standard deviation is less than a preset standard deviation threshold. If it is less, then based on the target sparsity rate, the semantic relevance score matrix is ​​sparsely filtered according to the standard deviation to determine the semantically relevant and semantically irrelevant positions. The preset standard deviation threshold can be set by those skilled in the art according to actual conditions, and this application does not impose specific limitations.

[0060] For example, in this embodiment, the target sparsity is set to 70%, meaning that the top 30% of positions with the highest scores are retained as semantically relevant positions, and the remaining positions are considered semantically irrelevant positions. The semantic relevance score matrix in this embodiment is as follows: When the standard deviation is calculated to be approximately 0.0166, and the preset standard deviation threshold is 0.05 (this application does not impose specific limitations), that is, the standard deviation is less than the preset standard deviation threshold. At this time, the embodiment of this application can perform sparse filtering on the semantic relevance score matrix based on the target sparsity rate. That is, according to the score ranking result, the position corresponding to the highest score of 0.9 is determined as the semantically relevant position, and the remaining positions are determined as the semantically irrelevant positions.

[0061] In some embodiments, if the standard deviation is greater than or equal to a preset standard deviation threshold, the semantic relevance score matrix can be divided based on a target threshold to determine the semantically relevant and semantically irrelevant positions.

[0062] For example, in the embodiments of this application, the semantic relevance score matrix is... In this case, the calculated standard deviation is approximately 0.35, and the preset standard deviation threshold is 0.05 (this application does not impose specific limitations). That is, the standard deviation is greater than or equal to the preset standard deviation threshold. In this case, the embodiment of this application can divide the semantic relevance score matrix based on the target threshold, and the target threshold is 0.5 (this application does not impose specific limitations). The positions of elements in the semantic relevance score matrix that are greater than or equal to the target threshold are determined as semantically relevant positions, and the positions of elements that are less than the target threshold are determined as semantically irrelevant positions, thereby obtaining semantically relevant positions and semantically irrelevant positions.

[0063] The embodiments of this application adaptively switch the screening strategy according to the dispersion (standard deviation) of the semantic relevance score distribution: sparsity rate is used when the distribution is concentrated, and learning threshold is used when the distribution is dispersed, which can improve the robustness, accuracy and adaptability of attention matrix screening.

[0064] In step S104, the mask matrix corresponding to the input word vector matrix is ​​determined based on the semantic relevance score matrix, semantically irrelevant positions, and semantically relevant positions.

[0065] In some embodiments, this application can construct and generate a mask matrix corresponding to the input word vector matrix based on the matrix size of the semantic relevance score matrix and the determined semantically relevant and semantically irrelevant positions.

[0066] This application's embodiments convert the semantic relevance screening results into a structured mask to shield and constrain the attention mechanism, thereby improving the model's ability to focus on effective semantic relationships, reducing computational load, and enhancing robustness.

[0067] Optionally, in one embodiment of this application, determining the mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, semantically irrelevant positions, and semantically relevant positions includes: modifying the elements corresponding to the semantically irrelevant positions in the semantic relevance score matrix to a first target value, and modifying the elements corresponding to the semantically relevant positions in the semantic relevance score matrix to a second target value; and determining the mask matrix based on the first target value, the second target value, the semantically irrelevant positions, and the semantically relevant positions.

[0068] It is understood that, in the embodiments of this application, the first target value can be... The second target value can be 0, and can be set by those skilled in the art according to the actual situation. This application does not impose any specific limitations.

[0069] In some embodiments, the elements corresponding to semantically irrelevant positions in the semantic relevance score matrix can be modified to a first target value, and the elements corresponding to semantically relevant positions can be modified to a second target value, thereby determining the corresponding mask matrix.

[0070] For example, the mask matrix generated in the embodiments of this application can be: , in, This represents the mask matrix.

[0071] Optionally, in one embodiment of this application, the method further includes: based on the mask matrix, calculating the accuracy of the large language model under the corresponding semantic task; if the accuracy is less than a preset accuracy threshold, then based on the query vector and key vector, regenerating the mask matrix until the accuracy is greater than or equal to the preset accuracy threshold, so as to obtain the regenerated mask matrix.

[0072] In some embodiments, the currently generated mask matrix can be applied to a large language model to perform semantic-related tasks and calculate the accuracy of the model output. If the accuracy is less than a preset accuracy threshold, the mask matrix is ​​regenerated based on the query vector and key vector, as well as the backpropagation algorithm, until the accuracy is greater than or equal to the preset accuracy threshold, thereby obtaining the regenerated mask matrix.

[0073] The embodiments of this application iteratively optimize the mask matrix through task accuracy feedback, enabling the mask to adaptively match semantic tasks and improve the focusing ability and task accuracy of large language models.

[0074] Optionally, in one embodiment of this application, the method further includes: obtaining a causal mask matrix adapted to the self-attention mechanism of the Transformer decoder; determining the superposition method of the causal mask matrix and the mask matrix; and determining the composite mask matrix corresponding to the input word vector matrix based on the causal mask matrix, the mask matrix, and the superposition method.

[0075] It is understood that, in the embodiments of this application, the causal mask matrix can be understood as a lower triangular matrix used for the self-attention mechanism to ensure that each position can only focus on the current position and the sequence before it (i.e., causal relationship), thus preventing future information leakage.

[0076] The superposition method may include, but is not limited to, taking the maximum value element by element, taking the minimum value by superposition, adding elements by element, and adding through a sliding window. The specific method can be set by those skilled in the art according to the actual situation, and this application does not impose any specific restrictions.

[0077] In some embodiments, this application can obtain a causal mask matrix adapted to the self-attention mechanism of the Transformer decoder, determine the superposition method of the causal mask matrix and the semantic mask matrix, and then generate a composite mask matrix corresponding to the input word vector matrix based on the causal mask matrix, the semantic mask matrix and the superposition method.

[0078] For example, embodiments of this application may use a mask matrix. Causal mask matrix When the superposition method is to take the minimum value, the corresponding composite mask matrix is ​​obtained, which can be expressed as: .

[0079] This application's embodiments, while inheriting the temporal validity of the Transformer's native causal mask, superimpose the semantic sparsity of a semantic mask to mask irrelevant historical terms, such as... Figure 2 As shown, the Transformer attention mechanism in related technologies uses a fixed lower triangle mask, where all historical words are visible to the current word, making it impossible to distinguish semantic relevance. However, the embodiments of this application further filter based on semantic relevance on the basis of the fixed lower triangle mask, retaining only the visibility of semantically relevant historical words and performing secondary masking on irrelevant historical words, thus achieving more refined attention constraints.

[0080] It can be understood that, in the embodiments of this application, before applying the composite mask matrix to the Softmax calculation, the model is guided to establish attention connections only between lexical units that simultaneously satisfy temporal logical validity and semantic relevance, thereby reducing the amount of attention computation, improving the model's ability to focus on effective semantics, and enhancing robustness.

[0081] The working principle of the method for generating the large language model mask matrix proposed in this application will be introduced below with reference to several specific embodiments.

[0082] Example 1: in, Figure 3 This is a flowchart illustrating the working principle of a method for generating a large language model mask matrix according to an embodiment of this application.

[0083] Step S301: Obtain the input word vector matrix of the Transformer architecture, and determine the corresponding query vector and key vector based on the input word vector matrix.

[0084] In this embodiment, the corresponding query vector and key vector can be generated based on the input word vector matrix through linear mapping.

[0085] Step S302: Calculate the dimension-reduced query vector and the dimension-reduced key vector based on the dimension-reduced matrix.

[0086] In this embodiment, a dimensionality reduction matrix is ​​used for dimensionality reduction projection to generate a corresponding dimensionality reduction query vector. and dimensionality reduction key vector .

[0087] Step S303: Calculate the semantic relevance score matrix.

[0088] In this application embodiment, calculation can be performed. and The vector dot product generates a semantic relevance score matrix. .

[0089] Step S304: Calculate the standard deviation of each element in the semantic relevance score matrix.

[0090] Step S305: Detect whether the standard deviation is less than the preset standard deviation threshold.

[0091] If the value is less than the specified value, proceed to step S306; otherwise, proceed to step S307.

[0092] Step S306: Determine the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target sparsity.

[0093] Step S307: Determine the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold.

[0094] In this embodiment, the target threshold can be learned by a pre-built threshold learning model, as detailed above, and will not be repeated here.

[0095] Step S308: Modify the elements corresponding to the semantically irrelevant positions in the semantic relevance score matrix to the first target value, and modify the elements corresponding to the semantically relevant positions in the semantic relevance score matrix to the second target value.

[0096] The first target value can be The second target value can be 0, and this application does not impose any specific restrictions.

[0097] Step S309: Fuse and superimpose with the causal mask matrix to output a composite mask matrix.

[0098] In this embodiment of the application, the mask matrix can be used. Causal mask matrix When the superposition method is to take the minimum value, the corresponding composite mask matrix is ​​obtained, which can be expressed as: .

[0099] Example 2: in, Figure 4 This is a block diagram illustrating the overall architecture of a method for generating a large language model mask matrix according to an embodiment of this application.

[0100] Step S401: Input the word sequence.

[0101] In this embodiment of the application, the word sequence can be understood as the word sequence after the text has been segmented.

[0102] Step S402: Word embedding layer.

[0103] In this embodiment of the application, the word vector matrix can be obtained by mapping the word sequence into a vector through a word embedding layer and then adding position encoding.

[0104] Step S403: Generate a mask matrix.

[0105] In this embodiment, the input word vector matrix can be combined with... Figure 3 As shown, the mask matrix corresponding to the input word vector matrix is ​​generated.

[0106] Step S404: Obtain the causal mask matrix.

[0107] The causal mask matrix in this embodiment can be represented as: , Step S405: Superimpose masks to generate a composite mask matrix.

[0108] In this embodiment of the application, the mask matrix can be used. Causal mask matrix When the superposition method is to take the minimum value, the corresponding composite mask matrix is ​​obtained, which can be expressed as: .

[0109] Step S406: Multi-head attention calculation.

[0110] In this embodiment, based on the input word vector matrix and the composite mask matrix, the context feature vector is calculated through multi-head attention to generate the corresponding multi-head attention context feature vector.

[0111] Step S407: Residual connection and normalization.

[0112] In this embodiment, the multi-head attention context feature vector can be residually connected to the input word vector matrix, and then normalized through layer normalization to obtain the normalized feature vector.

[0113] Step S408: Feedforward Neural Network.

[0114] In this embodiment, a feedforward neural network can be used to independently perform nonlinear transformations (such as two-layer linear transformations and activation functions, which are not specifically limited in this application) on the features at each position of the normalized feature vector to further extract higher-order semantic information, enhance feature expression capabilities, and obtain the transformed feature vector.

[0115] Step S409: Output layer.

[0116] In this embodiment, the transformed feature vector can be mapped to a vocabulary probability distribution through Softmax using the output layer.

[0117] The method for generating a large language model mask matrix proposed in this application can calculate the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture, and then calculate the corresponding standard deviation. If the standard deviation is less than a preset standard deviation threshold, the semantically irrelevant and semantically relevant positions are determined based on the target sparsity rate; otherwise, the semantically irrelevant and semantically relevant positions are determined based on the target threshold, and then the corresponding mask matrix is ​​determined. This method can adapt to different score distribution scenarios, accurately distinguish between effective and redundant semantics, improve the rationality and applicability of the mask, and reduce the computational overhead and noise interference caused by irrelevant semantics. Therefore, it solves the problems in related technologies, such as high computational redundancy, excessive invalid full attention calculations, low inference efficiency, severe noise interference, fixed masks that distract attention, dilute core semantics, and affect generation and understanding, and lack of content awareness, fixed masks that cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making performance optimization difficult.

[0118] Next, with reference to the accompanying drawings, an apparatus for generating a large language model mask matrix according to an embodiment of this application is described.

[0119] Figure 5 This is a block diagram of a device for generating a large language model mask matrix according to an embodiment of this application.

[0120] like Figure 5 As shown, the large language model mask matrix generation device 10 is based on the Transformer architecture. The large language model mask matrix generation device 10 includes: a first calculation module 100, a second calculation module 200, a first determination module 300, and a second determination module 400.

[0121] The first calculation module 100 is used to calculate the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture.

[0122] The second calculation module 200 is used to calculate the standard deviation of each element in the semantic relevance score matrix.

[0123] The first determining module 300 is used to determine the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target sparsity rate when the standard deviation is less than a preset standard deviation threshold; otherwise, it determines the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold.

[0124] The second determining module 400 is used to determine the mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, semantically irrelevant positions, and semantically relevant positions.

[0125] Optionally, in one embodiment of this application, the first calculation module 100 includes: a first determining unit, a second determining unit, a first calculation unit, and a second calculation unit.

[0126] The first determining unit is used to determine the corresponding query vector and key vector based on the input word vector matrix.

[0127] The second determining unit is used to determine the corresponding dimensionality reduction matrix based on the query vector and the key vector.

[0128] The first computing unit is used to calculate the corresponding dimension-reduced query vector and dimension-reduced key vector based on the query vector, key vector, and dimension-reduced matrix.

[0129] The second calculation unit is used to calculate the semantic relevance score matrix based on the dimension-reduced query vector and the dimension-reduced key vector.

[0130] Optionally, in one embodiment of this application, the second determining module 400 includes: a modification unit and a third determining unit.

[0131] The modification unit is used to modify the elements corresponding to the semantically irrelevant positions in the semantic relevance score matrix to the first target value, and to modify the elements corresponding to the semantically relevant positions in the semantic relevance score matrix to the second target value.

[0132] The third determining unit is used to determine the mask matrix based on the first target value, the second target value, semantically irrelevant positions, and semantically relevant positions.

[0133] Optionally, in one embodiment of this application, it further includes: an acquisition module, a third determination module, and a fourth determination module.

[0134] The acquisition module is used to acquire the causal mask matrix adapted to the self-attention mechanism of the Transformer decoder.

[0135] The third determining module is used to determine the causal mask matrix and the superposition method of the mask matrix.

[0136] The fourth determination module is used to determine the composite mask matrix corresponding to the input word vector matrix based on the causal mask matrix, the mask matrix, and the superposition method.

[0137] Optionally, in one embodiment of this application, it further includes: a fifth determining module and a correcting module.

[0138] The fifth determining module is used to determine the initial threshold corresponding to the target threshold based on the input word vector matrix before determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold.

[0139] The correction module is used to obtain the training parameters and training results corresponding to the current training round of the large language model, and correct the initial threshold based on the training parameters and training results to obtain the corrected target threshold.

[0140] It should be noted that the explanation of the above-described method for generating a large language model mask matrix also applies to the apparatus for generating a large language model mask matrix in this embodiment, and will not be repeated here.

[0141] The large language model mask matrix generation device proposed in this application can calculate the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture, and then calculate the corresponding standard deviation. If the standard deviation is less than a preset standard deviation threshold, the semantically irrelevant and semantically relevant positions are determined based on the target sparsity. Otherwise, the semantically irrelevant and semantically relevant positions are determined based on the target threshold, and then the corresponding mask matrix is ​​determined. This device can adapt to different score distribution scenarios, accurately distinguish between effective and redundant semantics, improve the rationality and applicability of the mask, and reduce the computational overhead and noise interference caused by irrelevant semantics. Therefore, it solves the problems in related technologies, such as high computational redundancy, excessive invalid full attention calculations, low inference efficiency, severe noise interference, fixed masks that distract attention, dilute core semantics, and affect generation and understanding, and lack of content awareness, fixed masks that cannot be dynamically adjusted, resulting in a mismatch between computational overhead and semantic requirements, making performance optimization difficult.

[0142] Figure 6 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of this application. The electronic device may include: The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.

[0143] When the processor 602 executes the program, it implements the method for generating the large language model mask matrix provided in the above embodiments.

[0144] Furthermore, electronic devices also include: Communication interface 603 is used for communication between memory 601 and processor 602.

[0145] The memory 601 is used to store computer programs that can run on the processor 602.

[0146] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0147] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0148] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.

[0149] The processor 602 may be a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0150] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for generating a large language model mask matrix.

[0151] This application also provides a computer program product, including a computer program that, when executed, implements the above-described method for generating a large language model mask matrix.

[0152] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0153] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0154] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0156] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0157] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0158] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0159] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for generating a large language model mask matrix, characterized in that, The large language model is based on the Transformer architecture, and the method includes the following steps: Based on the input word vector matrix of the Transformer architecture, the corresponding semantic relevance score matrix is ​​calculated; Calculate the standard deviation of each element in the semantic relevance score matrix; If the standard deviation is less than a preset standard deviation threshold, then based on the target sparsity, the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix are determined; otherwise, based on the target threshold, the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix are determined. Based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions, the mask matrix corresponding to the input word vector matrix is ​​determined.

2. The method according to claim 1, characterized in that, The calculation of the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture includes: The corresponding query vector and key vector are determined based on the input word vector matrix; Based on the query vector and the key vector, determine the corresponding dimensionality reduction matrix; Based on the query vector, the key vector, and the dimensionality reduction matrix, calculate the corresponding dimensionality reduction query vector and dimensionality reduction key vector; The semantic relevance score matrix is ​​calculated based on the reduced-dimensional query vector and the reduced-dimensional key vector.

3. The method according to claim 1, characterized in that, The step of determining the mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions includes: The elements corresponding to the semantically irrelevant positions in the semantic relevance score matrix are modified to the first target value, and the elements corresponding to the semantically relevant positions in the semantic relevance score matrix are modified to the second target value; The mask matrix is ​​determined based on the first target value, the second target value, the semantically irrelevant position, and the semantically relevant position.

4. The method according to claim 1, characterized in that, Also includes: Obtain the causal mask matrix that adapts to the self-attention mechanism of the Transformer decoder; Determine the superposition method of the causal mask matrix and the mask matrix; Based on the causal mask matrix, the mask matrix, and the superposition method, the composite mask matrix corresponding to the input word vector matrix is ​​determined.

5. The method according to claim 1, characterized in that, Before determining the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold, the method further includes: Based on the input word vector matrix, determine the initial threshold corresponding to the target threshold; Obtain the training parameters and training results corresponding to the current training round of the large language model, and based on the training parameters and training results, correct the initial threshold to obtain the corrected target threshold.

6. A device for generating a large language model mask matrix, characterized in that, The large language model is based on the Transformer architecture, wherein the device includes: The first calculation module is used to calculate the corresponding semantic relevance score matrix based on the input word vector matrix of the Transformer architecture. The second calculation module is used to calculate the standard deviation of each element in the semantic relevance score matrix; The first determining module is used to determine the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target sparsity rate when the standard deviation is less than a preset standard deviation threshold; otherwise, it determines the semantically irrelevant and semantically relevant positions in the semantic relevance score matrix based on the target threshold. The second determining module is used to determine the mask matrix corresponding to the input word vector matrix based on the semantic relevance score matrix, the semantically irrelevant positions, and the semantically relevant positions.

7. The apparatus according to claim 6, characterized in that, The first computing module includes: The first determining unit is used to determine the corresponding query vector and key vector based on the input word vector matrix; The second determining unit is used to determine the corresponding dimensionality reduction matrix based on the query vector and the key vector; The first calculation unit is used to calculate the corresponding dimension-reduced query vector and dimension-reduced key vector based on the query vector, the key vector, and the dimension-reduced matrix; The second calculation unit is used to calculate the semantic relevance score matrix based on the dimensionality-reduced query vector and the dimensionality-reduced key vector.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the program to implement the method for generating a large language model mask matrix as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method for generating a large language model mask matrix as described in any one of claims 1-5.

10. A computer program product, characterized in that, Includes a computer program, which, when executed, is used to implement the method for generating a large language model mask matrix as described in any one of claims 1-5.

Images