Model data processing method and device, computer equipment and readable storage medium

Abstract

The invention relates to a model data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: converting text sequence feature data into a query vector, a key vector and a value vector through a decoder-only converter model; then multi-head attention vectors are determined; performing non-parameter shielding processing on the multi-head attention vector to obtain an attention score vector shielded by a pseudo attention parameter; performing normalization processing on the attention score vector through a normalization exponential function to obtain a normalized attention score vector; performing shielding score replacement processing on the normalized attention score vector to obtain a target attention score vector; and determining model output data of the text sequence features according to the product of the target attention score vector and the value vector. According to the method, the problem of unproportionate attention distribution can be solved in a non-parameter shielding manner, and meanwhile, absolute position information is coded in a self-attention processing process.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology and artificial intelligence (AI) technology, machine learning (ML) technology has emerged. Machine learning is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and pre-trained learning. Pre-trained models are the latest development in deep learning, integrating all of the above techniques.

[0003] The decoder-transformer model is a widely used model that can be applied to fields such as natural language processing (NLP). However, the architecture of the transformer model has problems such as over-focus and relative position encoding failing to capture enough absolute position information, which affects the quality of the model's output data, fails to guarantee model processing efficiency, and can easily lead to a waste of computer processing resources during the model processing. Summary of the Invention

[0004] Therefore, it is necessary to provide a model data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the model processing efficiency of the converter, in order to address the above-mentioned technical problems.

[0005] Firstly, this application provides a model data processing method, including:

[0006] The text sequence feature data is transformed into query vectors, key vectors, and value vectors using a decoder-only converter model.

[0007] Multi-head attention processing is performed based on the query vector and the key vector to obtain a multi-head attention vector;

[0008] The multi-head attention vector is subjected to non-parametric masking to obtain an attention score vector masked by pseudo-attention parameters;

[0009] The attention score vector is normalized by a normalized exponential function to obtain a normalized attention score vector.

[0010] The normalized attention score vector is subjected to masking score replacement processing to obtain the target attention score vector;

[0011] The model output data for the text sequence features is determined by multiplying the target attention score vector and the value vector.

[0012] Secondly, this application also provides a model data processing apparatus, comprising:

[0013] The vector transformation module is used to transform text sequence feature data into query vectors, key vectors, and value vectors through a decoder-only converter model.

[0014] A multi-head attention processing module is used to perform multi-head attention processing based on the query vector and the key vector to obtain a multi-head attention vector;

[0015] The nonparametric masking processing module is used to perform nonparametric masking processing on the multi-head attention vector to obtain the attention score vector masked by the pseudo attention parameters;

[0016] The normalization processing module is used to normalize the attention score vector using a normalization exponential function to obtain a normalized attention score vector.

[0017] The score replacement module is used to perform masking score replacement processing on the normalized attention score vector to obtain the target attention score vector;

[0018] The data output module is used to determine the model output data of the text sequence features based on the product of the target attention score vector and the value vector.

[0019] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0020] The text sequence feature data is transformed into query vectors, key vectors, and value vectors using a decoder-only converter model.

[0021] Multi-head attention processing is performed based on the query vector and the key vector to obtain a multi-head attention vector;

[0022] The multi-head attention vector is subjected to non-parametric masking to obtain an attention score vector masked by pseudo-attention parameters;

[0023] The attention score vector is normalized by a normalized exponential function to obtain a normalized attention score vector.

[0024] The normalized attention score vector is subjected to masking score replacement processing to obtain the target attention score vector;

[0025] The model output data for the text sequence features is determined by multiplying the target attention score vector and the value vector.

[0026] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0027] The text sequence feature data is transformed into query vectors, key vectors, and value vectors using a decoder-only converter model.

[0028] Multi-head attention processing is performed based on the query vector and the key vector to obtain a multi-head attention vector;

[0029] The multi-head attention vector is subjected to non-parametric masking to obtain an attention score vector masked by pseudo-attention parameters;

[0030] The attention score vector is normalized by a normalized exponential function to obtain a normalized attention score vector.

[0031] The normalized attention score vector is subjected to masking score replacement processing to obtain the target attention score vector;

[0032] The model output data for the text sequence features is determined by multiplying the target attention score vector and the value vector.

[0033] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0034] The text sequence feature data is transformed into query vectors, key vectors, and value vectors using a decoder-only converter model.

[0035] Multi-head attention processing is performed based on the query vector and the key vector to obtain a multi-head attention vector;

[0036] The multi-head attention vector is subjected to non-parametric masking to obtain an attention score vector masked by pseudo-attention parameters;

[0037] The attention score vector is normalized by a normalized exponential function to obtain a normalized attention score vector.

[0038] The normalized attention score vector is subjected to masking score replacement processing to obtain the target attention score vector;

[0039] The model output data for the text sequence features is determined by multiplying the target attention score vector and the value vector.

[0040] The aforementioned model data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product first transform text sequence feature data into query vectors, key vectors, and value vectors using a decoder-only converter model. Then, during the model's attention processing, multi-head attention processing is performed based on the query and key vectors to obtain a multi-head attention vector. Next, non-parametric masking is applied to this multi-head attention vector, resulting in an attention score vector masked by pseudo-attention parameters. Non-parametric masking replaces traditional causal masking operations, using pseudo-attention parameters to mask the attention score vector. This pseudo-attention score is used to allocate excessive attention. The attention score vector is then normalized using a normalized exponential function, and the masking score is replaced, thus maintaining the causal decoding characteristics of the non-parametric masking process and ensuring no information leakage. Simultaneously, by introducing pseudo-attention scores, the result of the normalization operation is no longer a right random matrix, allowing for the encoding of absolute positional information during normalization. Finally, the model output data for the text sequence features is determined based on the product of the target attention score vector and the value vector, completing the attention processing process of the decoder-only converter model. This application introduces additional pseudo-attention scores during the attention calculation process using non-parametric masking. These pseudo-attention scores are used to address excessive attention allocation, thus resolving the problem of disproportionate attention distribution. Simultaneously, non-parametric masking ensures that the result of normalization exponent processing is not a right random matrix, thereby encoding absolute position information during the normalization process. This solves the problem of inaccurate encoding of absolute position information, guarantees the effectiveness of attention output data, improves the processing performance of the converter model, and ultimately ensures model processing efficiency and improves the resource utilization efficiency of the computer. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is an application environment diagram of the model data processing method in one embodiment;

[0043] Figure 2This is a flowchart illustrating a model data processing method in one embodiment;

[0044] Figure 3 This is a schematic diagram of comparing a causal masking matrix with a nonparametric masking processing matrix in one embodiment;

[0045] Figure 4 This is a flowchart illustrating the non-parametric masking process in one embodiment.

[0046] Figure 5 This is a flowchart illustrating the self-attention processing flow in one embodiment;

[0047] Figure 6 This is a schematic diagram of the mask matrix and pseudo-attention score matrix in one embodiment;

[0048] Figure 7 This is a schematic diagram of the masking score replacement process in one embodiment;

[0049] Figure 8 This is a flowchart of a model data processing method in one embodiment;

[0050] Figure 9 This is a graph illustrating the difference in perplexity performance of non-parametric occlusion in one embodiment.

[0051] Figure 10 This is a schematic diagram illustrating the perplexity performance and model extrapolation effect of nonparametric occlusion in one embodiment;

[0052] Figure 11 This is a structural block diagram of a model data processing device in one embodiment;

[0053] Figure 12 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0055] This application relates to the field of artificial intelligence (AI). AI is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making functions. AI technology is a comprehensive discipline involving a wide range of fields, including both hardware and software technologies. Basic AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large models or foundational models, can be widely applied to downstream tasks in various AI directions after fine-tuning. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning. This application specifically relates to Natural Language Processing (NLP) and machine learning technologies within AI.

[0056] Natural Language Processing (NLP) is a crucial area within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP deals with natural language—the language people use in daily life—and is closely related to linguistics; it also involves computer science and mathematics. Pre-trained models, a key technique for model training in artificial intelligence, evolved from Large Language Models (LLM). After fine-tuning, these LLM models can be widely applied to downstream tasks. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0057] The model data processing method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104, or it can be located in the cloud or on another network server. Terminal 102 can submit data processing requests, while server 104 is equipped with a data processing model built on a decoder-only converter model. Upon receiving the data processing request from terminal 102, the server 104 can complete the data processing through the decoder-only converter model. This data processing includes attention-related processing. During attention-related calculations, server 104 obtains the query vector, key vector, and value vector from the decoder-only converter model; performs multi-head attention processing based on the query and key vectors to obtain a multi-head attention vector; performs non-parametric masking on the multi-head attention vector to obtain an attention score vector masked by pseudo-attention parameters; normalizes the attention score vector using a normalized exponential function to obtain a normalized attention score vector; performs masking score replacement processing on the normalized attention score vector to obtain a target attention score vector; and obtains the attention output data of the converter model based on the product of the target attention score vector and the value vector. Finally, the result of the data processing is obtained based on the attention output data and returned to terminal 102. The terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, and projection devices. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0058] In one exemplary embodiment, such as Figure 2 As shown, a model data processing method is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 211. Wherein:

[0059] Step 201: Transform the text sequence feature data into query vector, key vector, and value vector using a decoder-only converter model.

[0060] Among them, the decoder-only transformer model refers to a decoder-only model implemented solely as a transformer. A decoder-only model is a type of decoder where the model only contains the decoder part. When applied to natural language processing tasks, decoder-only models excel at text generation because information can only flow from left to right, effectively generating new vocabulary in an autoregressive manner. The transformer model, on the other hand, is a neural network model based on a self-attention mechanism, primarily used for processing sequential data, particularly excelling in natural language processing tasks such as machine translation, text generation, and speech recognition. The core of the transformer model is the self-attention mechanism, which allows the model to focus on different positions in a sequence when encoding or decoding it. This mechanism assigns different weights to each position in the sequence by calculating attention weights, and then outputs these weighted position vectors. The computation process of the self-attention mechanism includes three steps: calculating attention weights, calculating weighted position vectors, and outputting the weighted position vectors. These steps work together to enable the transformer model to effectively capture long-range dependencies when processing sequential data. Text sequence feature data is the target data input to the model for processing. The model output data can be obtained by processing the text sequence feature data. The query vector, key vector, and value vector refer to the three input representation vectors in the self-attention process. These three vectors are obtained from the text sequence feature data through linear transformation. In the self-attention mechanism, based on the query vector Q, a weight distribution is obtained by calculating the similarity between the query vector and all key vectors K, which is used to weight and sum the associated value vector V.

[0061] For example, when staff at terminal 102 need to perform data processing using a decoder-only model, such as completing a natural language processing task, they can submit a data request to server 104 equipped with a decoder-only converter model. Server 104 then extracts task-related text sequence feature data from the request and uses this data to call the decoder-only converter model for data processing. Specifically, this application applies to the attention processing process within the decoder-only converter model. During this process, the decoder-only converter model can extract query vectors, key vectors, and value vectors from the input text sequence feature data, and then uses these three vectors as a basis for self-attention calculations. In one embodiment, terminal 102 provides word embedding feature vectors as text sequence feature data. The decoder-only converter model on server 104 then transforms the word embedding feature vectors into query vectors, key vectors, and value vectors during the self-attention processing.

[0062] Step 203: Perform multi-head attention processing based on the query vector and key vector to obtain the multi-head attention vector.

[0063] Multi-Head Attention is mainly used to process the query vector Q and the key vector K. It can divide the hidden state vector into multiple heads to form multiple sub-semantic spaces, allowing the model to focus on information in different dimensions of the semantic space (or to focus on information from different aspects), resulting in a multi-head vector output.

[0064] For example, during self-attention processing, the converter model may excessively focus its attention on its own position when encoding information at the current location, thus reducing the model's expressive power. Therefore, multi-head attention can also be used to process the query vector and value vector during attention processing. In multi-head attention processing, the query vector and key vector can be split into multiple groups, and then self-attention processing can be performed on them. Then, the results of each group of self-attention are concatenated and subjected to a linear transformation to obtain the final output multi-head attention vector.

[0065] Step 205: Perform non-parametric masking on the multi-head attention vector to obtain the attention score vector masked by the pseudo attention parameters.

[0066] The non-parametric masking process is an improvement in the attention processing of existing solutions. Non-parametric masking is a parameter-free method that addresses the aforementioned limitations by improving the causal mask. The pseudo-attention parameter is primarily used to balance the attention distribution, thus preventing excessive attention. The non-parametric masking process balances the attention distribution by introducing pseudo-attention parameter values. Furthermore, during non-parametric masking, absolute positional information can be encoded by gradually decreasing the mask ratio, thereby enhancing model performance while maintaining its extrapolation capabilities.

[0067] For example, in the architecture of a decoder-only converter model, the property that the normalized exponential function (softmax function) requires all attention scores to be non-zero and sum to 1 can cause the model to over-focus on specific tags and locations in some cases. Furthermore, this relative positional encoding cannot capture sufficient absolute positional information. Therefore, this application solves this problem by using an attention mask to perform attention masking before the normalized exponential function processing. This application modifies the implementation of the causal mask, converting the padding values ​​before the normalization operation to a linear decay mode without affecting its original functionality, and then clearing them again after the normalization operation. This solves both of the aforementioned problems of the converter model without affecting the causal decoding characteristics of the model. In one embodiment, the matrix of a traditional causal mask can be referred to as... Figure 3 As shown in the top left, when applied to a decoder-only model, causal masking is mainly used to make the model focus only on the tags preceding the corresponding tag while masking subsequent tags. The matrix for the improved non-parametric masking processing in this application can be found by referring to... Figure 3 As shown in the lower right corner, by converting the padding values ​​in the upper right part of the matrix to a linear decay pattern and then clearing them again after normalization, the two problems mentioned above, which exist only in the converter model, are solved simultaneously without affecting the causal decoding characteristics of the model.

[0068] Nonparametric masking constructs a nonparametric masked attention score matrix containing pseudo-attention scores. When the current attention head is less dependent on its preceding context, it can selectively store unnecessary attention values ​​on these pseudo-attention scores. Assuming a fixed sequence length of n per row, each row will have ni pseudo-attention scores for excessive attention allocation. This addresses the issue that the severity of over-attention gradually increases with the growth of the decoder model sequence, i.e., more pseudo-attention values ​​at the beginning of the sequence. The calculation of the attention score vector during nonparametric masking can be specifically described by the formula A ⊙ C + P, where A is the input multi-head attention vector, C is a mask matrix, and P is a matrix containing pseudo-attention scores.

[0069] Step 207: Normalize the attention score vector using a normalized exponential function to obtain a normalized attention score vector.

[0070] Step 209: Perform masking score replacement processing on the normalized attention score vector to obtain the target attention score vector.

[0071] The normalization exponential function, also known as the softmax function, "compresses" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), ensuring that each element falls within the range (0,1) and the sum of all elements is 1. This function is often used in multi-class classification problems. In self-attention processing, it is mainly used to normalize unnormalized attention weights. However, the softmax function's requirement that all attention scores be non-zero and sum to 1 can lead to the model over-focusing on specific labels and locations in certain situations, thus affecting the model's performance. Masking score replacement refers to replacing the scores that need to be masked with 0.

[0072] For example, after obtaining the attention score vector, it can be normalized using the normalization exponential function softmax. This normalization process normalizes the unnormalized attention weights in the attention score matrix. For the formula A ⊙ C + P, the normalization exponential processing can be represented as Softmax(A ⊙ C + P), and the result is the normalized attention score vector. Then, a masking score replacement process can be performed on the normalized attention score vector. This process corresponds to the masking process and can be specifically represented as Softmax(A ⊙ C + P) ⊙ C. Here, the attention score vector is masked with pseudo-attention scores inside Softmax, while C outside Softmax replaces the scores that need to be masked with 0. Therefore, nonparametric masking still maintains the characteristics of causal decoding, ensuring that information is not leaked from subsequent words. In one embodiment, the complete nonparametric masking process of this application can be referred to... Figure 4 As shown, for the input multi-head attention vector A, the multi-head attention vector A is processed by the mask matrix C and the pseudo-attention score matrix P. First, the multi-head attention vector A is multiplied by the mask matrix C, then the result of the multiplication is added to the pseudo-attention score matrix P, and the result is input into the softmax function for processing. Then, the result of the processing is multiplied by the mask matrix C again, thus completing the masking score replacement process and obtaining the target attention score vector.

[0073] Step 211: Determine the model output data of the text sequence features based on the product of the target attention score vector and the value vector.

[0074] For example, after obtaining the target attention score vector, the target attention score vector and the value vector can be multiplied to obtain the output result of the attention processing. Finally, in the decoder-only converter model, corresponding calculations can be performed based on the attention output result. In one embodiment, the solution of this application is applied to natural language processing. In this case, the decoder-only converter model is a GPT (Generative Pre-Trained) model, which can perform text generation processing related to natural language processing. At this time, after determining the attention output data for the input text sequence feature data, text generation processing can be performed based on the attention output data to obtain the final model output data. The complete attention processing flow can be referred to... Figure 5As shown, the input text sequence feature data is first transformed to obtain densely labeled query vector, key vector and value vector. Then, the key vector of the query vector is processed by multi-head attention (MHA). The result is input into the softmax function for normalization under non-parametric masking. Finally, the output result is obtained based on the processed target attention score vector and the densely represented value vector.

[0075] The nonparametric masking proposed in this application serves as a direct replacement for causal masks, exhibiting high compatibility with the native architecture of the converter model and supporting various positional encoding, attention optimization, and extrapolation techniques. Furthermore, it allows for the construction of versions suitable for inference by adding suffix markers to the end of the sequence to adapt to variations in sequence length, thereby reducing computational costs during inference.

[0076] The aforementioned model data processing method first transforms the text sequence feature data into query vectors, key vectors, and value vectors using a decoder-only converter model. Then, during the model's attention processing, multi-head attention is first performed based on the query and key vectors to obtain a multi-head attention vector. Next, non-parametric masking is applied to this multi-head attention vector, resulting in an attention score vector masked by pseudo-attention parameters. Non-parametric masking replaces traditional causal masking operations, using pseudo-attention parameters to mask the attention score vector. This pseudo-attention score is used to distribute excessive attention. The attention score vector is then normalized using a normalized exponential function, and the masking score is replaced, thus preserving the causal decoding characteristics of the non-parametric masking process and ensuring no information leakage. Furthermore, by introducing pseudo-attention scores, the result of the normalization operation is no longer a right random matrix, allowing for the encoding of absolute positional information during the normalization process. Finally, the model output data for the text sequence features is determined based on the product of the target attention score vector and the value vector, completing the attention processing process of the decoder-only converter model. This application introduces additional pseudo-attention scores during the attention calculation process using non-parametric masking. These pseudo-attention scores are used to address excessive attention allocation, thus resolving the problem of disproportionate attention distribution. Simultaneously, non-parametric masking ensures that the result of normalization exponent processing is not a right random matrix, thereby encoding absolute position information during the normalization process. This solves the problem of inaccurate encoding of absolute position information, guarantees the effectiveness of attention output data, improves the processing performance of the converter model, and ultimately ensures model processing efficiency and improves the resource utilization efficiency of the computer.

[0077] In an exemplary embodiment, step 205 includes: masking the multi-head attention vector using a masking matrix to obtain a masked attention matrix; and finally, obtaining the attention score vector masked by the pseudo-attention parameters based on the sum of the masked attention matrix and the pseudo-attention score matrix.

[0078] The mask matrix is ​​a matrix composed of 0s and 1s. In Natural Language Processing (NLP), sentences are of varying lengths, but because sentences are often divided into multiple batches for training, shorter sentences are padded with zeros at the end. The mask matrix indicates which data is real and which is padding. In the mask matrix, 1 represents real data; 0 represents padding. Pseudoattention is a technique used in neural networks to replace standard self-attention mechanisms. This application primarily addresses the problem of disproportionate attention allocation by introducing additional pseudoattention scores to address excessive attention allocation.

[0079] For example, in the scheme of this application, during non-parametric masking processing, the disproportionate attention allocation problem and the inability of relative position encoding to capture sufficient absolute position information are mainly solved by combining the mask matrix and the pseudo-attention score matrix. After obtaining the multi-head attention vector, the multi-head attention vector is first masked using the mask matrix. For the input multi-head attention vector A and the mask matrix C, the masking process can be represented as A ⊙ C. After the masking process is completed and the mask attention matrix that actually needs to be processed is identified, the pseudo-attention score matrix is ​​then used to mask the score of the mask attention matrix. For the pseudo-attention score matrix P, the processing can be represented as A ⊙ C+P, using the pseudo-attention in the pseudo-attention score matrix to mask the attention score matrix. In one embodiment, the mask matrix C and the pseudo-attention score matrix P can be referenced... Figure 6 As shown, the upper right corner of the mask matrix C is 0, indicating that this part is masked. For the pseudo-attention score matrix, its upper triangular portion introduces additional pseudo-attention scores. Each row has pseudo-attention scores for excessive attention allocation. γ in the pseudo-attention score matrix represents hyperparameters, which are manually set before the algorithm runs to control the model's behavior and performance. These parameters are not learned from the data during training but need to be predefined. The choice of hyperparameters has a significant impact on the model's training speed, convergence, capacity, and generalization ability. The final masking score replacement process can be referred to... Figure 7 As shown, a mask matrix is ​​used to replace some of the scores that need to be masked with 0. In this embodiment, the original attention score matrix is ​​masked with pseudo-attention scores inside the normalized exponential function using a mask matrix and a pseudo-attention score matrix, thereby effectively ensuring the model's performance.

[0080] In an exemplary embodiment, the method further includes: determining the vector structure of the multi-head attention vector input; and constructing a pseudo-attention score matrix based on the vector structure.

[0081] For example, before performing nonparametric masking using the pseudo-attention score matrix, it is necessary to construct the pseudo-attention scores. During construction, the vector structure of the multi-head attention vector input can be determined first. Then, a pseudo-attention score matrix consistent with the vector structure is constructed. Specifically, the construction process of the pseudo-attention score matrix can be as follows: first, determine the number of rows and row sequence length of the vector structure; then, determine the position and number of pseudo-attention scores in each row of the pseudo-attention score matrix based on the row sequence length; finally, construct the pseudo-attention score matrix according to the position and number of pseudo-attention scores in each row. In one embodiment, assuming the sequence length of each row in the vector structure is fixed at n, it can be determined that each row will have ni pseudo-attention scores for excessive attention allocation when setting the pseudo-attention scores, where i represents the position, i.e., the i-th row, meaning there are more pseudo-attention values ​​at the beginning of the sequence. This solves the problem that the severity of the over-attention problem gradually increases with the growth of the decoder model sequence. For a 4x4 vector structure, the matrix has 4 rows and the row sequence length is 4. In this case, 3 pseudo-attention scores can be set for the first row, 2 pseudo-attention scores for the second row, and 1 pseudo-attention score for the third row. In this embodiment, the position and number of pseudo-attention scores in each row are determined by the row sequence length, and then a pseudo-attention score matrix is ​​constructed, thereby effectively realizing non-parametric occlusion processing and ensuring the accuracy and efficiency of the processing.

[0082] In one embodiment, step 209 includes: performing masking score replacement processing on the normalized attention score vector using a mask matrix to obtain the target attention score vector.

[0083] For example, the masking score replacement process is similar to the non-parametric masking process. After masking the attention score matrix with pseudo-attention scores inside the Softmax, the masked scores need to be replaced with 0. Specifically, this can be achieved by performing masking score replacement on the normalized attention score matrix using a mask matrix. For the non-parametric masking process of A ⊙ C + P, the corresponding masking score replacement process can be represented as Softmax(A ⊙ C + P) ⊙ C. Here, the attention score vector is masked with pseudo-attention scores inside the Softmax, and C outside the Softmax replaces the scores that need to be masked with 0 again. Therefore, non-parametric masking still maintains the characteristics of causal decoding, ensuring that information is not leaked from subsequent words. It also includes the construction process of the mask matrix, specifically determining the vector structure of the multi-head attention vector input; and then constructing the corresponding mask matrix based on the vector structure. For example, for... Figure 4The 3*3*n multi-head attention vector input shown can be used to construct a corresponding 3*3 structure mask matrix. During construction, the number of rows and row sequence length of the vector structure matrix must first be determined; then, based on the number of rows and row sequence length, a mask matrix with a progressively decreasing mask ratio is constructed. For example... Figure 6 The 3x3 mask matrix C shown has a mask portion in the upper right corner. The first row has two mask 0s, and the second row has one mask. By setting a gradually decreasing mask ratio, after nonparametric masking, it can be ensured that the result processed by the normalized exponential function is not a right random matrix, thus encoding absolute position information in the softmax stage and solving the problem of inaccurate encoding of absolute position information. Furthermore, nonparametric masking has been theoretically proven to recover the absolute position information of the input sequence. This means that even without explicit position encoding, the model can infer the absolute position information of each tag through the attention score encoded by nonparametric masking. In this embodiment, by constructing a mask matrix with a gradually decreasing mask ratio and then using the mask matrix to replace the masking score, absolute position information can be effectively encoded in the normalization stage of the normalized exponential function, solving the problem of inaccurate encoding of absolute position information. This improves the model's output processing performance and thus increases the processing efficiency of text data processed by the converter model.

[0084] In one embodiment, the text sequence feature data includes summary text sequence feature data. The method further includes: obtaining a text summarization processing request; extracting the summary text sequence feature data from the text summarization processing request. Step 211 further includes: determining self-attention output data based on the product of the target attention score vector and the value vector; performing text generation processing based on the self-attention output data to obtain a text summary.

[0085] A text summarization request refers to a request to summarize specified text data. Text summarization refers to the process of extracting, summarizing, or refining key information from text or a collection of texts using various technologies to summarize and present the main content or gist of the original text. The core issue of text summarization is how to extract key information from redundant, unstructured long texts to form a concise and fluent summary.

[0086] For example, when text summarization processing is required, the terminal can submit a text summarization processing request to a server equipped with the decoder-only converter model of this application, and then the server processes the text summarization task based on the request. Specifically, the solution of this application is a generative text summarization method. After receiving a text summarization processing request, it first extracts the summary text sequence feature data representing the text information from the text summarization processing request, and inputs the summary text sequence feature data as basic data into the decoder-only converter model. During the model data processing, the decoder-only converter model performs self-attention processing in a non-parametric masking manner to obtain self-attention output data. Then, based on the self-attention output data, subsequent text generation-related processing operations are performed to generate the text summarization result information. In another embodiment, this application can also be applied to the field of machine translation. After obtaining a text translation processing request, the translation text sequence feature data in the text translation processing request can also be used as basic data. Then, the decoder-only converter model performs corresponding processing, and finally outputs self-attention output data and performs corresponding text translation processing to obtain the text translation result. In this embodiment, by applying the method to text summarization and machine translation, the model performance can be improved, and accurate text summarization content and machine translation results can be output more efficiently.

[0087] This application also provides an application scenario, which is illustrated by taking the above-mentioned model data processing method as an example. The model data processing method specifically includes:

[0088] When a user wants to extract summary information from specified text content, text summarization can be performed using a GPT-type large language model, which consists of a decoder-only converter model. Users can build a decoder-only converter model based on the model data processing method described in this application, and then construct a GPT large language model using multiple converter models. The user can then input the text information to be summarized as text sequence feature data into the large language model, which will then perform the text summarization. This application mainly relates to the process of performing self-attention processing on the input text sequence feature data using a decoder-only converter model. Based on the self-attention processing, text generation can be performed to obtain the text summary content.

[0089] The complete process of achieving self-attention processing through non-parametric occlusion in this application can be found in [reference]. Figure 8 As shown, Figure 8 In (1), the difference between the pseudo-attention score matrix and the traditional causal masking matrix used in this application is shown. Figure 8As shown in (2), during the processing, the input text sequence feature data is transformed into query vector Q, key vector K, and value vector V. Then, Q and K undergo multi-head attention processing, and the result A is imported into the softmax function. The non-parametric masking process can be referred to Figure 8 As shown in Figure (3), the input multi-head attention vector is first multiplied by the mask matrix C, then the result of the multiplication is added to the pseudo-attention score matrix, and then softmax processing is performed, which is represented as softmax(A ⊙ C + P). The matrices C and P can be referred to Figure 8 (5). After normalization, the masking score replacement process is still required through the mask matrix. This process can be referred to Figure 8 As shown in (4), the final output is processed with the value vector V to output the final self-attention processing result. For the text processing results, please refer to... Figure 8 As shown in (6). In addition, this application also includes the construction process of pseudo attention score matrix and mask matrix. Both matrices correspond to the input data. For pseudo attention score matrix, each row will have ni pseudo attention scores for over-attention allocation, where n is the sequence length and i is the number of rows. For mask matrix, the mask matrix will set a gradually decreasing mask ratio to encode absolute position information.

[0090] like Figure 9 As shown, the perplexity (PPL) performance of models with and without nonparametric masking was compared on the Wikitext-103 and MiniPile datasets using models of different sizes (from 71M to 1.4B parameters). On the Pile dataset, the 1.4B parameter model was pre-trained and its performance was evaluated on downstream tasks such as LAMBAD, PIQA, ARC-Easy, ARC-Challenge, OpenbookQA, and Winogrande. Validation results and extrapolated model performance can be found in [reference needed]. Figure 10 The results showed that the model using nonparametric masking improved both perplexity and downstream task performance, further validating its effectiveness.

[0091] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0092] Based on the same inventive concept, this application also provides a model data processing apparatus for implementing the model data processing method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model data processing apparatus embodiments provided below can be found in the limitations of the model data processing method described above, and will not be repeated here.

[0093] In one exemplary embodiment, such as Figure 11 As shown, a model data processing device is provided, comprising:

[0094] The vector transformation module 1102 is used to transform text sequence feature data into query vectors, key vectors, and value vectors through a decoder-only converter model.

[0095] The multi-head attention processing module 1104 is used to perform multi-head attention processing based on the query vector and the key vector to obtain a multi-head attention vector.

[0096] The nonparametric masking processing module 1106 is used to perform nonparametric masking processing on the multi-head attention vector to obtain the attention score vector masked by the pseudo attention parameters.

[0097] The normalization processing module 1108 is used to normalize the attention score vector through a normalization exponential function to obtain a normalized attention score vector.

[0098] The score replacement module 1110 is used to perform masking score replacement processing on the normalized attention score vector to obtain the target attention score vector.

[0099] The data output module 1112 is used to determine the model output data of text sequence features based on the product of the target attention score vector and the value vector.

[0100] In one embodiment, the non-parametric masking processing module 1106 is specifically used to: mask the multi-head attention vector using a masking matrix to obtain a masked attention matrix; and obtain the attention score vector masked by the pseudo-attention parameters based on the sum of the masked attention matrix and the pseudo-attention score matrix.

[0101] In one embodiment, a pseudo-attention matrix construction module is further included, which is used to: determine the vector structure of the multi-head attention vector input; and construct a pseudo-attention score matrix based on the vector structure.

[0102] In one embodiment, the pseudo-attention matrix construction module is specifically used to: determine the number of rows and row sequence length of the vector structure matrix; determine the position and number of pseudo-attention scores in each row of the pseudo-attention score matrix based on the row sequence length; and construct the pseudo-attention score matrix according to the position and number of pseudo-attention scores in each row.

[0103] In one embodiment, the score replacement module 1110 is specifically used to: perform masking score replacement processing on the normalized attention score matrix through a mask matrix.

[0104] In one embodiment, a mask matrix construction module is further included, which is used to: determine the vector structure of the multi-head attention vector input; and construct a mask matrix based on the vector structure.

[0105] In one embodiment, the matrix construction module is specifically used to: determine the number of rows and row sequence length of the vector structure matrix; and construct a mask matrix with a mask ratio decreasing row by row based on the number of rows and row sequence length of the vector structure matrix.

[0106] In one embodiment, the text sequence feature data includes summary text sequence feature data. The apparatus further includes a request acquisition module, configured to: acquire a text summarization processing request; and extract the summary text sequence feature data from the text summarization processing request. The data output module is specifically configured to: determine self-attention output data based on the product of the target attention score vector and the value vector; and perform text generation processing based on the self-attention output data to obtain a text summary.

[0107] In one embodiment, the text sequence feature data includes translated text sequence feature data; the apparatus further includes a request acquisition module, configured to: acquire a text translation processing request; and extract the translated text sequence feature data from the text translation processing request. The data output module is specifically configured to: determine self-attention output data based on the product of the target attention score vector and the value vector; and perform text translation processing based on the self-attention output data to obtain the text translation result.

[0108] Each module in the aforementioned model data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0109] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data related only to the decoder-converter model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a model data processing method.

[0110] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0111] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0112] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0113] In one embodiment, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above method embodiments.

[0114] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model data processing method, characterized in that, The method includes: The text sequence feature data is transformed into query vectors, key vectors, and value vectors using a decoder-only converter model. Multi-head attention processing is performed based on the query vector and the key vector to obtain a multi-head attention vector; The multi-head attention vector is subjected to non-parametric masking to obtain an attention score vector masked by pseudo-attention parameters; The attention score vector is normalized by a normalized exponential function to obtain a normalized attention score vector. The normalized attention score vector is subjected to masking score replacement processing to obtain the target attention score vector; The model output data for the text sequence features is determined by multiplying the target attention score vector and the value vector.

2. The method according to claim 1, characterized in that, The non-parametric masking process performed on the multi-head attention vector to obtain the attention score vector masked by pseudo-attention parameters includes: The multi-head attention vector is masked using a masking matrix to obtain a masked attention matrix; Based on the sum of the masked attention matrix and the pseudo-attention score matrix, the attention score vector masked by the pseudo-attention parameters is obtained.

3. The method according to claim 2, characterized in that, The method further includes: Determine the vector structure of the multi-head attention vector input; A pseudo-attention score matrix is ​​constructed based on the aforementioned vector structure.

4. The method according to claim 3, characterized in that, The construction of the pseudo-attention score matrix based on the vector structure includes: Determine the number of matrix rows and the row sequence length of the vector structure; The position and number of pseudo-attention scores in each row of the pseudo-attention score matrix are determined based on the row sequence length. Construct a pseudo-attention score matrix based on the position and number of pseudo-attention scores in each row.

5. The method according to claim 1, characterized in that, The process of performing masking score replacement on the normalized attention score vector to obtain the target attention score vector includes: The normalized attention score matrix is ​​masked and replaced using a mask matrix.

6. The method according to claim 5, characterized in that, The method further includes: Determine the vector structure of the multi-head attention vector input; Construct a mask matrix based on the vector structure.

7. The method according to claim 6, characterized in that, The process of constructing the mask matrix based on the vector structure includes: Determine the number of matrix rows and the row sequence length of the vector structure; Based on the number of rows and row sequence length of the vector structure, a mask matrix with a mask ratio decreasing row by row is constructed.

8. The method according to any one of claims 1 to 7, characterized in that, The text sequence feature data includes summary text sequence feature data; The method further includes: Get text summary processing request; Extract the summary text sequence feature data from the text summary processing request; The model output data for determining the text sequence features based on the product of the target attention score vector and the value vector includes: The self-attention output data is determined based on the product of the target attention score vector and the value vector. Text generation processing is performed based on the self-attention output data to obtain a text summary.

9. The method according to any one of claims 1 to 7, characterized in that, The text sequence feature data includes translated text sequence feature data; The method further includes: Obtain a text translation processing request; Extract the translation text sequence feature data from the text translation processing request; The model output data for determining the text sequence features based on the product of the target attention score vector and the value vector includes: The self-attention output data is determined based on the product of the target attention score vector and the value vector. Text translation is performed based on the self-attention output data to obtain the text translation result.

10. A model data processing device, characterized in that, The device includes: The vector transformation module is used to transform text sequence feature data into query vectors, key vectors, and value vectors through a decoder-only converter model. A multi-head attention processing module is used to perform multi-head attention processing based on the query vector and the key vector to obtain a multi-head attention vector; The nonparametric masking processing module is used to perform nonparametric masking processing on the multi-head attention vector to obtain the attention score vector masked by the pseudo attention parameters; The normalization processing module is used to normalize the attention score vector using a normalization exponential function to obtain a normalized attention score vector. The score replacement module is used to perform masking score replacement processing on the normalized attention score vector to obtain the target attention score vector; The data output module is used to determine the model output data of the text sequence features based on the product of the target attention score vector and the value vector.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 9.

Images