Method and device for determining response text based on dialogue text and electronic equipment

Abstract

The invention discloses a method and device for determining a response text based on a dialogue text and electronic equipment, and relates to the technical field of language processing. The method comprises the steps of determining current to-be-reasoned information according to a dialogue text of a user and a current reasoning round of a neural network model; determining a floating point type value vector feature matrix corresponding to the current to-be-reasoned information, wherein the floating point type value vector feature matrix comprises multiple columns of features; quantizing the floating point type value vector feature matrix based on a quantization parameter corresponding to each column of features in the multiple columns of features to obtain a fixed point type value vector feature matrix; obtaining a current reasoning result based on the current to-be-reasoned information, the fixed-point type value vector feature matrix and at least one historical fixed-point type value vector feature matrix; and determining a response text corresponding to the dialogue text according to the current reasoning result. According to the technical scheme, the problem that an existing method needs to consume a large number of computing resources and storage resources and cannot be applied to terminal equipment with limited resources can be solved.

Description

Technical Field

[0001] The present disclosure relates to the field of language processing technology, and in particular to a method, device and electronic device for determining a response text based on a conversation text. Background Art

[0002] Dialogue response is an important application area of ​​large language models, which can simulate human dialogue, provide instant feedback, and solve user problems. In actual application scenarios, large language models are usually used to generate response text. Large language models refer to neural network models with very large parameter scales, usually including billions or even hundreds of billions of parameters, which can understand and generate natural language text, thereby realizing dialogue response functions. Summary of the invention

[0003] When generating response text, large language models consume a lot of computing and storage resources in the inference stage to perform intensive floating-point operations during the inference process, as well as to store and read the data generated during the inference process. Therefore, large language models are usually deployed in high-performance computing environments such as cloud servers, and their deployment in terminal devices with more limited computing and storage resources such as in-vehicle terminals and mobile phones is greatly restricted. In other words, the current response text generation method is not suitable for resource-constrained terminal devices.

[0004] In order to solve the above technical problems, the present disclosure provides a method, device and electronic device for determining a response text based on a conversation text, which can reduce the amount of calculation of a large language model in the inference stage, especially reduce the amount of floating-point calculation, so that the large language model can be deployed in terminal devices with limited computing resources such as vehicle terminals and mobile phones. In this way, the response text can be determined in a terminal device with limited resources.

[0005] In a first aspect of the present disclosure, a method for determining a response text based on a conversation text is provided, comprising: determining current information to be inferred based on a user's conversation text and a current reasoning round of a neural network model; determining a floating-point value vector feature matrix corresponding to the current information to be inferred, the floating-point value vector feature matrix comprising multiple columns of features; quantizing the floating-point value vector feature matrix based on quantization parameters corresponding to each column of features in the predetermined multiple columns of features, to obtain a fixed-point value vector feature matrix corresponding to the current information to be inferred; obtaining a current reasoning result corresponding to the current information to be inferred based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in a key-value cache space; and determining a response text corresponding to the conversation text based on the current reasoning result.

[0006] According to a second aspect of the present disclosure, there is provided a device for determining a response text based on a conversation text, comprising: an input module configured to determine current information to be inferred based on the conversation text of the user and the current reasoning round of the neural network model; a linearization module configured to determine a floating-point value vector feature matrix corresponding to the current information to be inferred, the floating-point value vector feature matrix comprising multiple columns of features; a quantization module configured to quantize the floating-point value vector feature matrix based on a quantization parameter corresponding to each column of features in a predetermined multiple columns of features, to obtain a fixed-point value vector feature matrix corresponding to the current information to be inferred; an output module configured to obtain a current reasoning result corresponding to the current information to be inferred based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space; and a response module configured to determine the response text corresponding to the conversation text based on the current reasoning result.

[0007] According to a third aspect of the present disclosure, an electronic device is provided, which includes: one or more processors, and a memory; the memory stores computer instructions; when the computer instructions are executed by the processor, the processor executes the method for determining a response text based on a conversation text provided in the first aspect.

[0008] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions. When the computer program instructions are executed by a processor, the processor executes the method for determining a response text based on a conversation text provided in the first aspect.

[0009] A fifth aspect of the present disclosure provides a computer program product. When instructions in the computer program product are executed by a processor, the method for determining a response text based on a conversation text provided in the first aspect is performed.

[0010] Based on the method for determining the response text based on the dialogue text provided by the present disclosure, by predetermining the quantization parameters corresponding to each column of features, using the quantization parameters corresponding to each column of features, quantizing the fixed-point value vector feature matrix, so that each feature on the same column in the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix corresponds to the same quantization parameter, and then the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix can be directly calculated in fixed points. Since the present disclosure can reduce the number of quantization and dequantization operations performed by the large language model in the inference stage, and reduce the amount of floating-point calculations in the inference process, the large language model can be deployed in terminal devices with limited computing resources such as vehicle terminals and mobile phones, avoiding the deployment of large language models on cloud servers, ensuring that the terminal device can quickly determine the response text under the condition of limited computing resources, and improving the efficiency of executing dialogue response operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a schematic diagram of an attention structure provided by an exemplary embodiment of the present disclosure.

[0012] Figure 2 It is the first simplified workflow diagram of the attention structure provided by an exemplary embodiment of the present disclosure.

[0013] Figure 3 Schematic diagram of two quantization methods provided by an exemplary embodiment of the present disclosure.

[0014] Figure 4 It is a flowchart of a method for determining a response text based on a conversation text provided by an exemplary embodiment of the present disclosure.

[0015] Figure 5 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0016] Figure 6 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0017] Figure 7 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0018] Figure 8 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0019] Fig. 9 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0020] Fig.10 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0021] Fig.11 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0022] Fig.12 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0023] Fig.13 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0024] Fig.14is a schematic diagram of a matrix multiplication layer provided by an exemplary embodiment of the present disclosure.

[0025] Fig.15 It is a flowchart of a method for determining a response text based on a conversation text provided by another exemplary embodiment of the present disclosure.

[0026] Fig.16 A schematic diagram of matrix multiplication is provided for an exemplary embodiment of the present disclosure.

[0027] Fig.17 An exemplary embodiment of the present disclosure provides a device for determining a response text based on a conversation text.

[0028] Fig.18 A device for determining a response text based on a conversation text is provided as another exemplary embodiment of the present disclosure.

[0029] Fig.19 A structural diagram of an electronic device provided as an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.

[0031] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0032] Application Overview

[0033] Dialogue response is an important application area of ​​the big language model. It can process input text, extract semantic information, and generate output text through big language processing. Through such a design, it can simulate human dialogue and provide instant feedback.

[0034] The neural network model needs to perform intensive floating-point operations in the inference stage, as well as store and read the data generated in the inference stage, so it needs to consume certain computing resources. The neural network model can be deployed in electronic devices such as servers, data centers, and terminal devices (for example, mobile phones, vehicle-mounted terminals). Therefore, the computing resources required for the neural network model can be provided by the above electronic devices. For example, the electronic device may include at least one processor. The processor may include a central processing unit, a neural network processor, etc., which is used to provide the computing resources required for the neural network model.

[0035] As a neural network model with a very large parameter scale, the large language model usually includes billions or even hundreds of billions of parameters. Therefore, the large language model needs to consume a lot of computing resources in the inference stage. Taking the attention structure contained in the large language model as an example, in the inference stage, the attention structure can calculate the relationship between the current information to be inferred and all the historical information to be inferred that is currently input into the large language model, and dynamically adjust the attention weight, so as to effectively focus on the information most relevant to the context of the current information to be inferred, so that the large language model can generate output related to the context of the current information to be inferred, and improve the performance of the large language model in tasks such as natural language processing. In order to improve the accuracy of the neural network model, the parameters in the large language model are usually floating-point parameters. Therefore, in the inference stage, the attention structure needs to perform a large number of floating-point calculations. For example, matrix multiplication calculations are performed on related floating-point parameters, which requires a large amount of floating-point computing resources. This makes it difficult to deploy large language models in terminal devices with limited computing resources such as mobile phones and vehicle-mounted terminals.

[0036] In order to solve the above technical problems, the embodiments of the present disclosure provide a method, device and electronic device for determining a response text based on a conversation text, which can reduce the amount of calculation of a large language model in the reasoning stage, so that the large language model can be deployed in terminal devices with limited computing resources such as vehicle-mounted terminals and mobile phones. In addition, it can also reduce the amount of data access to the large language model during the reasoning process, reduce the demand for computing power and storage resources of the neural network model, and speed up the reasoning speed of the large language model. In this way, the large language model can be deployed in a terminal device with limited resources, and the response text can be determined in the terminal device with limited resources.

[0037] Example Neural Network Model

[0038] Figure 1 It is a schematic diagram of an attention structure provided by an exemplary embodiment of the present disclosure.

[0039] In one example, Figure 1 The attention structure shown is applied to the neural network model based on the transformer architecture, for example, in the large language model based on the transformer architecture. Figure 1 As shown, from the input side to the output side of the Attention structure, the Attention structure includes: a first matrix multiplication (MatMul) layer, a scaling (Scale) layer, a masking (Mask) layer, a normalization (Softmax) layer, and a second matrix multiplication layer.

[0040] The current information to be inferred input into the neural network model can be transformed into three different matrices through three different linear transformations, namely, the query matrix Q, the key matrix K and the value matrix V, hereinafter referred to as the matrix Q, the matrix K and the matrix V. Among them, the matrix Q represents the query information of the current information to be inferred, the matrix K represents the index information of the current information to be inferred, and the matrix V represents the actual information of the current information to be inferred. Generally speaking, the parameters in the current information to be inferred, the matrix Q, the matrix K and the matrix V are all floating-point parameters.

[0041] In the matrices Q, K, and V, each row parameter in the matrix, that is, each vector in the matrix, corresponds to a token of the current input, and each column parameter corresponds to a feature channel. Usually, each vector in the matrix has a specific dimension d k , for example k =64, this dimension d k It is the number of characteristic channels of matrix Q, matrix K and matrix V, and also the number of columns of matrix Q, matrix K and matrix V.

[0042] A token is the smallest unit used for neural network processing. For example, in the field of natural language processing (NLP), a token is the smallest unit obtained after segmenting a text. A token may correspond to a word, a subword, or a character.

[0043] Matrix Q, matrix K and matrix V can be used as inputs of the Attention structure. The Attention structure can calculate the correlation parameters between the current information to be inferred and other information to be inferred based on the matrix Q and the matrix K, determine the attention weight matrix corresponding to the matrix V based on the calculation results of the correlation parameters, and perform weighted calculation on the matrix V based on the attention weight matrix to obtain the attention output of the Attention structure.

[0044] Combine the following Figure 1 , further explain the functions of each layer in the Attention structure. Among them:

[0045] The first matrix multiplication layer is used to multiply the matrix Q and the matrix K by T Perform matrix multiplication to obtain the matrix QK used to represent the correlation between the current information to be inferred and other information to be inferred T .

[0046] The scaling layer is used to use the matrix QK T Scale the parameter values ​​in to avoid the matrix QK T The parameter value in is too large, which is convenient for the subsequent matrix QK T Perform a normalization operation. For example, the scaling ratio can be dimension dk The square root of , after scaling, we get the matrix

[0047] The masking layer is used to transform the matrix Do a matrix dot multiplication with a mas matrix K of the same dimension and parameter value 0 or 1 to get QK T The parameters corresponding to the future information to be inferred are blocked, so that the neural network model only performs inference output based on the parameters of the historical information to be inferred.

[0048] The normalization layer is used to scale the matrix obtained through the softmax function. Normalize it and The parameter values ​​in are converted into weight values ​​ranging from 0 to 1.

[0049] The second matrix multiplication is used to normalize the matrix Perform matrix multiplication with matrix V to obtain the attention output Attention(Q,K,V) corresponding to the current input.

[0050] Right now:

[0051]

[0052] Next, the attention output corresponding to the current information to be inferred is processed by residual connection and layer normalization, so that the current inference result corresponding to the current information to be inferred can be obtained.

[0053] It should be noted that in tasks such as text generation and time series prediction, the reasoning stage of the neural network model can use autoregression to take the reasoning results of the previous stage of the neural network model as the information to be inferred in the current stage of the neural network model.

[0054] Figure 2 It is the first simplified workflow diagram of the attention structure provided by an exemplary embodiment of the present disclosure.

[0055] Next, combine Figure 2 , an exemplary illustration of the workflow of the Attention structure is given.

[0056] Taking the text generation task as an example, the initial input dialogue text of the neural network model is "How are you?", and the corresponding response text is generated using a neural network containing an Attention structure. The workflow of the Attention structure includes the prefill stage and the decoding stage. Figure 2 Each row in the figure corresponds to a round of reasoning, where the first row is the prefill stage, and the second and third rows are the decode stage.

[0057] In the prefill phase, the neural network model can encode the conversation text and obtain the corresponding matrices Q, K, and V through three different linear transformations, and initialize the cache space KV cache to store the matrices K and V in the KV cache. The Attention structure performs the above matrix multiplication, normalization, and other calculations on the matrices Q, K, and V to obtain the attention output corresponding to the first token "I". The attention output corresponding to token "I" is processed by residual connection and layer normalization, and the inference result corresponding to token "I" can be obtained.

[0058] Next, we enter the decoding phase, and perform multiple rounds of reasoning in sequence through autoregression to gradually generate the attention output corresponding to each token. In each round of reasoning, the reasoning result corresponding to the previous token is used as the input information to be reasoned into the neural network to obtain the attention output and reasoning result corresponding to the next token.

[0059] For example, the inference result corresponding to "I" is used as the information to be inferred by the neural network. Through three different linear transformations, the corresponding matrices Q, K, and V are obtained, and the matrices K and V corresponding to "I" are stored in the KV cache. The historically cached matrices K and V are read from the KV cache, and the matrix K corresponding to "I" is concatenated with the historically cached matrix K in the KV cache to obtain a new matrix K. The matrix V corresponding to "I" is concatenated with the historically cached matrix V in the KV cache to obtain a new matrix V. Then, based on the matrix Q, the new matrix K, and the new matrix V, the attention output and inference result corresponding to the second token "am" are obtained.

[0060] Next, the Attention structure uses the inference result corresponding to "am" as the information to be inferred, and obtains the corresponding matrices Q, K, and V through three different linear transformations, and stores the matrices K and V corresponding to "am" in the key-value cache KV cache. Read the historically cached matrices K and V from the KV cache, concatenate the matrix K corresponding to "am" with the historically cached matrix K in the KV cache to obtain a new matrix K, and concatenate the matrix V corresponding to "am" with the historically cached matrix V in the KV cache to obtain a new matrix V. Then, based on the matrix Q, the new matrix K, and the new matrix V, the attention output and inference result corresponding to the third token "fine" are obtained.

[0061] In this way, the response text "I am fine" corresponding to the dialogue text "How are you?" can be obtained through the neural network.

[0062] It can be understood that by setting up the KV cache to cache the matrix K and the matrix V, the Attention structure does not need to repeatedly calculate the matrix K and the matrix V corresponding to the historical input during each round of reasoning. Therefore, it can greatly reduce the amount of calculation of the neural network in the reasoning stage, improve the reasoning efficiency of the neural network, and then improve the efficiency of executing dialogue response operations.

[0063] Figure 3 Schematic diagram of two quantization methods provided by an exemplary embodiment of the present disclosure.

[0064] Next, combine Figure 3 , two existing quantization methods are explained.

[0065] Currently, the quantization schemes of KV Cache are usually static per-tensor quantization or dynamic per-tensor / per-token quantization. Among them, static quantization uses a calibration set to estimate the numerical distribution of activation values ​​during inference to determine the quantization parameters. This quantization method has a large error. Dynamic quantization includes dynamic per-tensor quantization and dynamic per-token quantization. Among them, Figure 3 As shown, in per-tensor quantization, all parameters in the matrix are combined to calculate a quantization parameter (such as Figure 3 In per-tensor quantization, the int8 parameters corresponding to the per-tensor quantization are calculated together, and the same quantization parameter is used for quantization. Therefore, the quantization granularity is coarse, resulting in a large quantization loss. In per-token quantization, the parameters of each token in the matrix are used together to calculate the quantization parameter corresponding to the token (such as Figure 3 The quantization parameters corresponding to each token are dynamically calculated during each inference. As a result, a large number of quantization and dequantization operations are performed before the KV-related matrix multiplication (matmul) step during the inference process, which requires more floating-point computing power. This makes it difficult to deploy large language models in terminal devices with limited computing resources such as in-vehicle terminals and mobile phones.

[0066] Exemplary Methods

[0067] Figure 4 is a flow chart of a method for determining a response text based on a conversation text provided by an exemplary embodiment of the present disclosure. This embodiment can be applied to a terminal device with limited resources, and can also be used in a cloud server, such as Figure 4 As shown, the following steps are included:

[0068] Step 41, determining the current information to be inferred based on the user's conversation text and the current inference round of the neural network model.

[0069] In this step, Figure 2 In the first round of reasoning (i.e., the prefill stage), the current information to be reasoned is the dialogue text input by the user, such as "How are you?" The dialogue text may be the text directly input by the user, or may be the text recognized from the video or voice input by the user, which is not limited in the embodiments of the present disclosure.

[0070] exist Figure 2 In other reasoning rounds shown (i.e., the decoding stage), the current information to be inferred is the reasoning result of the neural network model in the previous round. For example, in the second round of reasoning, the current information to be inferred is the reasoning result of the neural network model in the first round of reasoning, such as the reasoning result corresponding to token "I"; in the third round of reasoning, the current information to be inferred is the reasoning result of the neural network model in the second round of reasoning, such as the reasoning result corresponding to token "am", and so on.

[0071] It should be noted that Figure 2 This is only an exemplary embodiment. Other embodiments may include more or fewer reasoning rounds, which is not limited in the embodiments of the present disclosure.

[0072] Step 42, determining a floating-point value vector feature matrix corresponding to the current information to be inferred, wherein the floating-point value vector feature matrix includes multiple columns of features.

[0073] In this step, the floating-point value vector feature matrix is ​​the matrix V obtained by the neural network model by linearly transforming the current information to be inferred. The dimension of the matrix V can be N×d k , where N is the length of the current input sequence, that is, the number of tokens corresponding to the current information to be inferred, d k The number of feature channels, that is, the number of columns of matrix V.

[0074] It should be noted that the dimensions of the matrix V can be different in different reasoning rounds. For example, Figure 2 In the first round of reasoning (i.e., the prefill phase), if the current information to be reasoned is “How are you?”, which contains 4 tokens, then N=4. Figure 2 It can be seen that the matrix V has 4 rows at this time. Figure 2 In the second round of reasoning (i.e., the decoding stage), if the current information to be reasoned is the reasoning result corresponding to token "i", then N = 1, that is, the floating-point value vector feature matrix can be of dimension 1×d k The matrix V. From Figure 2 It can be seen that at this time the matrix V has 1 row.

[0075] Step 43, based on the quantization parameters corresponding to each column of the predetermined multiple columns of features, quantize the floating-point value vector feature matrix to obtain the fixed-point value vector feature matrix corresponding to the current information to be inferred.

[0076] Quantization refers to converting a floating-point parameter into a data type with lower precision, such as converting it into a fixed-point parameter.

[0077] In this step, multiple columns of features of the floating-point value vector feature matrix (e.g., d k Each column of features in the feature set has its corresponding quantization parameter. The quantization parameter corresponding to each column of features can be predetermined before the neural network model is deployed or the inference process begins, and remains unchanged during the subsequent inference process, that is, the quantization parameter corresponding to each column of features is a static parameter.

[0078] Specifically, based on the quantization parameters corresponding to each column of features, each column of features (i.e., each column of floating-point parameters) in the floating-point value vector feature matrix V is quantized to convert each column of floating-point parameters into fixed-point parameters (i.e., integer parameters), and the fixed-point value vector feature matrix V corresponding to the floating-point value vector feature matrix is ​​obtained. int .

[0079] In one implementation, the quantization parameters corresponding to each column of features may include at least a scaling ratio corresponding to each column of features. In this way, each column of floating-point parameters in the floating-point value vector feature matrix V may be mapped to a fixed-point parameter of a specified length based on the scaling ratio corresponding to each column of features.

[0080] The quantization parameters corresponding to each column of features may also include zero-point offset values ​​corresponding to each column of features. The zero-point offset value is used to accurately map the zero value of the floating-point parameter before quantization to the zero value of the fixed-point parameter after quantization.

[0081] Step 44, based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space, obtain the current inference result corresponding to the current information to be inferred.

[0082] In this step, during the inference process of the neural network model, the fixed-point value vector feature matrix obtained in each round of inference process can be cached in the memory of the electronic device as a historical fixed-point value vector feature matrix, such as stored in Figure 2In this way, in each round of reasoning, the Attention structure can directly read the historical fixed-point value vector feature matrix corresponding to the historical information to be reasoned from the KV Cache, avoiding repeated calculation of the historical fixed-point value vector feature matrix, greatly reducing the amount of calculation of the neural network model in the reasoning stage, thereby improving the calculation efficiency of the neural network model in the reasoning stage.

[0083] For example, Figure 2 Taking the second round of reasoning as an example, the current information to be reasoned is the reasoning result corresponding to token "i", and the historical information to be reasoned "How are you?" contains 4 tokens. Therefore, the KV cache caches the matrix V corresponding to "How are you?" int(How are you？) , the matrix V int(How are you？) The dimension is 4×d k Matrix V int(How are you？) That is the historical fixed-point value vector feature matrix.

[0084] The Attention structure can access memory and read the cached historical fixed-point value vector feature matrix, such as Figure 2 In the second round of reasoning shown in the figure, the dotted box corresponding to the matrix V is the cached historical fixed-point value vector feature matrix read from the KV cache. After that, the Attention structure can use the current information to be reasoned, the fixed-point value vector feature matrix corresponding to the current information to be reasoned, and the historical fixed-point value vector feature matrix to perform reasoning and obtain the current reasoning result corresponding to the current information to be reasoned.

[0085] Among them, the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix can be combined, and the current information to be inferred and the combined matrix can be used to perform reasoning operations to obtain the current reasoning result. The current information to be inferred can also be used to perform reasoning operations with the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix, and the obtained reasoning results can be combined to obtain the current reasoning result.

[0086] Step 45, determining the response text corresponding to the dialogue text according to the current reasoning result.

[0087] In this step, the current reasoning result is combined with the historical reasoning results of the previous reasoning rounds to obtain the response text corresponding to the dialogue text.

[0088] For example, Figure 2As shown, the user's dialogue text is "How are you?". In the first round of reasoning, the reasoning result corresponding to the token "I" is obtained; in the second round of reasoning, the reasoning result corresponding to the token "am" is obtained; in the third round of reasoning, the reasoning result corresponding to the token "fine" is obtained. At this time, if the reasoning is determined to be finished, the answer text corresponding to the dialogue text "How are you?" is determined to be "I am fine".

[0089] In view of the present disclosure, it should be noted that in the traditional quantization method, the quantization parameters corresponding to each column of features are not statically determined in advance, resulting in different scaling ratios of the features in the same column in the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix corresponding to the current information to be inferred. Therefore, it is necessary to first eliminate the difference between the scaling ratios of the features on the same column in the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix through inverse quantization calculation, and then perform subsequent matrix multiplication calculations to obtain the inference result.

[0090] The method for determining the response text based on the dialogue text provided by the embodiment of the present disclosure predetermines the quantization parameters corresponding to each column of features, and uses the quantization parameters corresponding to each column of features to quantize the fixed-point value vector feature matrix, so that each feature on the same column in the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix corresponds to the same quantization parameter, and then the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix can be directly calculated in fixed points. Since the present disclosure can reduce the number of quantization and dequantization operations performed by the neural network model in the inference stage, and reduce the amount of floating-point calculations in the inference process, the large language model can be deployed in terminal devices with limited computing resources such as vehicle terminals and mobile phones, avoiding the deployment of large language models on cloud servers, ensuring that the terminal device can quickly determine the response text under limited computing resources, and improving the efficiency of executing dialogue response operations.

[0091] like Figure 5 As shown in the above Figure 4 Based on the illustrated embodiment, step 44 may include the following steps:

[0092] Step 441 , determining a fixed-point query vector feature matrix corresponding to a floating-point query vector feature matrix corresponding to the current information to be inferred.

[0093] In this step, if Figure 2 In each round of reasoning shown in the figure, the floating-point query vector feature matrix is ​​the matrix Q obtained by the neural network model by linearly transforming the current information to be reasoned, and the matrix Q is quantized to obtain the matrix Q int , which is the fixed-point query vector feature matrix. Among them, the matrix Q intThe features in are fixed-point parameters. The specific implementation method will be described in the subsequent embodiments and will not be repeated here.

[0094] Step 442, determining the fixed-point key vector feature matrix corresponding to the floating-point key vector feature matrix corresponding to the current information to be inferred.

[0095] In this step, Figure 2 In each round of reasoning shown in the figure, the floating-point key vector feature matrix is ​​the matrix K obtained by the neural network model by linearly transforming the current information to be reasoned, and the matrix K is quantized to obtain the matrix K int , which is the fixed-point query vector feature matrix. Among them, the matrix K int The features in are fixed-point parameters.

[0096] Step 443, obtain a fixed-point attention parameter matrix corresponding to the dialogue text based on the fixed-point query vector feature matrix, the fixed-point key vector feature matrix, and at least one historical fixed-point key vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space.

[0097] In this step, the fixed-point attention parameter matrix is ​​determined based on the fixed-point key vector feature matrix and the fixed-point key vector feature matrix corresponding to the current information to be inferred, as well as the cached historical fixed-point key vector feature matrix. For the description of the historical fixed-point key vector feature matrix, please refer to the aforementioned historical fixed-point value vector feature matrix, which will not be repeated here.

[0098] Among them, the fixed-point attention matrix Y int It can be the correlation matrix Y used to characterize the correlation between the current information to be inferred and the historical information to be inferred int , fixed-point attention matrix Y int The parameters in are all fixed-point parameters. Fixed-point attention matrix Y int It can be an M×N dimensional matrix. In the prefill stage, M and N are equal to the number of tokens in the information to be inferred that is input into the neural network during the first round of reasoning. In the decode stage, M=1, and N is equal to the sum of the number of tokens in the historical information to be inferred and the current information to be inferred.

[0099] like Figure 2 The fixed-point attention parameter matrix can be expressed as QK shown in each reasoning round TIn the first round of reasoning (prefill stage), the fixed-point attention parameter matrix is ​​a 4×4 matrix, M=4, N=4; in the second round of reasoning (decode stage), the fixed-point attention parameter matrix is ​​a 1×5 matrix, M=1, N=5; in the third round of reasoning (decode stage), the fixed-point attention parameter matrix is ​​a 1×6 matrix, M=1, N=6. It should be noted here that Figure 2 This is only a simplified flowchart of an exemplary embodiment, so steps such as quantization are omitted. Figure 2 QK T It does not mean directly performing matrix multiplication on the transpose of the floating-point query vector feature matrix Q and the floating-point key vector feature matrix K.

[0100] Step 444, obtaining a current inference result based on the fixed-point attention parameter matrix, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix.

[0101] In this step, based on the fixed-point attention parameter matrix, a weighted operation is performed on the fixed-point value vector feature matrix and at least one historical fixed-point value vector feature matrix to obtain the current inference result.

[0102] The disclosed embodiment obtains a fixed-point key vector feature matrix and a fixed-point query vector feature matrix by quantizing the floating-point key vector feature matrix and the floating-point query vector feature matrix respectively. The fixed-point query vector feature matrix, the fixed-point key vector feature matrix, and the historical fixed-point key vector feature matrix are used to obtain a fixed-point attention parameter matrix, and then the fixed-point attention parameter matrix, the fixed-point value vector feature matrix, and the historical fixed-point value vector feature matrix are used to obtain the current reasoning result. On the basis of completing the reasoning operation, since the historical fixed-point key vector feature matrix and the historical fixed-point value vector feature matrix of the previous round of reasoning are cached in Figure 2 In the KV Cache shown in the figure, it is only necessary to calculate the fixed-point key vector feature matrix and the fixed-point value vector feature matrix corresponding to the current information to be inferred, which greatly reduces the amount of calculation of the neural network model in the inference stage and improves the inference efficiency of the neural network model, thereby improving the dialogue response efficiency and reducing the requirements for the computing power resources of the terminal device due to the inference operation.

[0103] like Figure 6 As shown in the above Figure 5 Based on the illustrated embodiment, step 444 may include the following steps:

[0104] Step 4441, dequantize the fixed-point attention parameter matrix to obtain the floating-point attention parameter matrix corresponding to the dialogue text.

[0105] In this step, the inverse quantization parameter can be determined based on the quantization parameter used when obtaining the fixed-point key vector feature matrix and the fixed-point query vector feature matrix. Based on the inverse quantization parameter, the fixed-point attention parameter matrix is ​​inversely quantized to obtain the floating-point attention parameter matrix. The achievable method of determining the inverse quantization parameter will be described in detail in the subsequent embodiments and will not be repeated here.

[0106] Among them, the floating point attention parameter matrix Y float It can be an M×N dimensional matrix. In the first round of reasoning, M and N are equal to the number of tokens in the information to be reasoned input into the neural network in the first round of reasoning. In other rounds of reasoning, M=1, and N is equal to the sum of the number of tokens in the historical information to be reasoned and the current information to be reasoned.

[0107] For example, the floating point attention parameter matrix Y float It can be calculated using the following formula:

[0108] Y float =Y int Y scale .

[0109] Among them, Y scale is the inverse quantization parameter.

[0110] Step 4442, normalize the floating-point attention parameter matrix to obtain a floating-point normalized attention parameter matrix.

[0111] In this step, the floating-point attention parameter matrix can be normalized by a softmax function, etc., and each parameter value in the floating-point attention parameter matrix can be converted into a weight value ranging from 0 to 1 to obtain a floating-point normalized attention parameter matrix. Among them, each weight value in the floating-point normalized attention parameter matrix indicates which parts of the current information to be inferred should be paid attention to by the neural network model when generating the current inference result. The higher the weight value, the more important the part corresponding to the weight value is, so the part is paid more attention to when generating the current inference result.

[0112] Step 4443, quantize the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix.

[0113] In this step, based on the quantization parameters of the floating-point normalized attention parameter matrix, each floating-point parameter in the floating-point normalized attention parameter matrix is ​​quantized, and the floating-point parameters are converted into fixed-point parameters to obtain a fixed-point normalized attention parameter matrix.

[0114] Step 4444, obtain the current inference result based on the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix.

[0115] In this step, based on the fixed-point normalized attention parameter matrix, an attention-based weighted operation is performed on the fixed-point value vector feature matrix and at least one historical fixed-point value vector feature matrix to obtain the current inference result.

[0116] The disclosed embodiment sequentially dequantizes, normalizes, and quantizes the fixed-point attention parameter matrix, and can perform normalization calculations on the premise of floating-point type to obtain a fixed-point normalized attention parameter matrix. Compared with the method of directly normalizing the fixed-point attention parameter matrix, the disclosed embodiment can obtain more accurate results. In addition, the disclosed embodiment maps each parameter value in the fixed-point attention parameter matrix to between 0 and 1 through normalization, and the sum of each parameter value is 1, so that the Attention structure can explain the importance of each part of the information to be inferred in a probabilistic manner, avoid interference from non-important information, and greatly improve the attention paid to important information in the information to be inferred.

[0117] like Figure 7 As shown in the above Figure 6 Based on the illustrated embodiment, step 4444 may include the following steps:

[0118] Step 1a, concatenating a fixed-point valued vector feature matrix and at least one historical fixed-point valued vector feature matrix to obtain a fixed-point valued vector concatenated feature matrix.

[0119] In this step, the Attention structure can access the KV Cache, read at least one historical fixed-point value vector feature matrix in the cache, and concatenate the fixed-point value vector feature matrix with the historical fixed-point value vector feature matrix to obtain a fixed-point value vector concatenation feature matrix corresponding to the current information to be inferred and the historical information to be inferred.

[0120] For example, Figure 2 Taking the second round of reasoning as an example, the current information to be reasoned is the reasoning result corresponding to token "i", and the fixed-point value vector feature matrix corresponding to the current information to be reasoned is the matrix V int(i) , the historical fixed-point value vector feature matrix is ​​the matrix Since the matrix V int Each row of corresponds to a token, so the matrix V int(i) and the matrix V int(How are you？) Concatenate in the row dimension, for example, in the matrix V int(How are you？) The last row of parameters is followed by the concatenated matrix V int(i) . The dimension is 5×d k The matrix Vint(How are you？i) , that is, the fixed-point value vector concatenation feature matrix. Among them, the matrix V int(How are you？i) The fifth row of parameters in is the matrix V int(i) Parameters in . It should be noted here that Figure 2 This is only a simplified flowchart of an exemplary embodiment, so steps such as quantization are omitted. Figure 2 In the second round of reasoning shown in the figure, one row is spliced ​​after the matrix V with four rows in the dotted box to obtain a new matrix V with five rows, which does not mean that the floating-point value vector feature matrix is ​​directly spliced.

[0121] It can be understood that in each round of reasoning, since a static quantization parameter is used for each column feature when the floating-point value vector feature matrix is ​​quantized, each parameter on the same column in the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix corresponds to the same quantization parameter. Furthermore, in the fixed-point value vector concatenation feature matrix, each parameter on the same column also corresponds to the same quantization parameter.

[0122] For example, the matrix V int(How are you？i) Each column in corresponds to a feature channel, and the matrix V int(How are you？i) Each column of parameters in corresponds to the same quantization parameter.

[0123] Step 1b, perform matrix multiplication on the fixed-point normalized attention parameter matrix and the fixed-point value vector concatenation feature matrix to obtain the current inference result.

[0124] In this step, the fixed-point normalized attention parameter matrix can be a weight matrix W containing fixed-point normalized attention parameters int Among them, the fixed-point normalized attention parameter matrix W int It can be an M×N dimensional matrix. In the first round of reasoning, M and N are equal to the number of tokens in the current information to be reasoned. In other rounds of reasoning, M=1, and N is equal to the sum of the number of tokens in the historical information to be reasoned and the current information to be reasoned.

[0125] For example, the fixed-point normalized attention parameter matrix can be composed of the matrix corresponding to the current information to be inferred It is obtained by normalization and quantization. In the above-mentioned embodiment, the method of obtaining the fixed-point normalized attention parameter matrix has been specifically described, which will not be repeated here.

[0126] In one implementation, a fixed-point calculation can be performed on the fixed-point normalized attention parameter matrix and the fixed-point value vector concatenation feature matrix, and based on the fixed-point calculation result, the inference result corresponding to the current information to be inferred can be obtained. For example, a matrix multiplication calculation can be performed on the fixed-point normalized attention parameter matrix and the fixed-point value vector concatenation feature matrix to obtain the attention output corresponding to the current information to be inferred, and then, based on the attention output corresponding to the current information to be inferred, the current inference result can be obtained. Exemplarily, the attention output corresponding to the current information to be inferred can be a dimension of 1×d k Fixed-point attention output matrix Attn int .

[0127] Taking the inference result corresponding to token “i” as an example, the fixed-point normalized attention parameter matrix is ​​the weight matrix W int(i) , the dimension is 1×5, and the fixed-point value vector concatenation feature matrix is ​​the matrix V int(How are you？i) , with dimensions of 5×d k . For the matrix W int(i) and the matrix V int(How are you？i) Perform matrix multiplication, that is, multiply the matrix W int(i) A row of parameters in the matrix V int(How are you？i) Each column parameter in is multiplied one by one, and the results are accumulated to obtain a dimension of 1×d k Fixed-point attention output matrix Attn int(am) .

[0128] Next, the attention output corresponding to the current information to be inferred is calculated, for example, the attention output matrix Attn int , after residual connection and layer normalization, the current inference result corresponding to the current information to be inferred can be obtained. The specific processing content depends on the structure of the neural network model and is not limited here.

[0129] In the disclosed embodiment, the fixed-point value vector splicing feature matrix is ​​characterized in that each parameter on the same column corresponds to the same quantization parameter. This feature makes it consistent with the calculation method of general matrix multiplication (GEMM). Therefore, the fixed-point normalized attention parameter matrix and the fixed-point value vector splicing feature matrix can be directly matrix multiplied to avoid additional inverse quantization calculations, thereby greatly improving the computational efficiency of matrix multiplication calculations in the inference stage. For example, in the fixed-point value vector splicing feature matrix, the parameters in the same column correspond to the same scaling ratio. Therefore, the fixed-point normalized attention parameter matrix W int The feature matrix concatenated with the fixed-point value vector can be directly multiplied. This can reduce the amount of floating-point calculations in the reasoning process, improve the reasoning efficiency of the neural network model, and further improve the efficiency of executing the dialogue response operation.

[0130] like Figure 8 As shown in the above Figure 6 Based on the illustrated embodiment, step 4443 may include the following steps:

[0131] Step 2a, based on the numerical range of each attention value in the floating-point normalized attention parameter matrix, determine the global quantization parameter shared by each parameter in the floating-point normalized attention parameter matrix.

[0132] In this step, the floating-point normalized attention parameter matrix can correspond to a global quantization parameter, that is, each parameter in the floating-point normalized attention parameter matrix (that is, each attention value) is quantized using this global quantization parameter, or in other words, the quantization parameters corresponding to each attention value in the floating-point normalized attention parameter matrix are the same.

[0133] Among them, the global quantization parameter can be a dynamic parameter, and the corresponding global quantization parameter in the floating-point normalized attention parameter matrix can be determined based on the parameter values ​​in the floating-point normalized attention parameter matrix.

[0134] Step 2b, based on the global quantization parameter, quantize the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix.

[0135] In this step, the floating-point normalized attention parameter matrix has a unified global quantization parameter. Based on the global quantization parameter, each floating-point parameter in the floating-point normalized attention parameter matrix is ​​quantized respectively to convert each floating-point parameter into a fixed-point parameter, and the fixed-point normalized attention parameter matrix W corresponding to the floating-point normalized attention parameter matrix is ​​obtained. int .

[0136] The disclosed embodiment uses global quantization parameters to quantize the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix. Compared with per-tensor quantization and per-token quantization, the disclosed embodiment adopts a global quantization method, which can reduce the amount of calculation in the quantization process of the floating-point normalized attention parameter matrix, improve the quantization efficiency of the floating-point normalized attention parameter matrix, and thus improve the efficiency of executing the dialogue response operation.

[0137] like Fig. 9 As shown in the above Figure 8 Based on the embodiment shown, step 2b may include the following steps:

[0138] Step 3a, based on the global quantization parameter, quantize the floating-point normalized attention parameter matrix with a preset quantization precision to obtain a fixed-point normalized attention parameter matrix corresponding to the preset quantization precision. The preset quantization precision is higher than the quantization precision corresponding to the fixed-point value vector feature matrix.

[0139] In this step, the fixed-point normalized attention parameter matrix has a corresponding preset quantization precision, and the floating-point normalized attention parameter matrix is ​​quantized with the preset quantization precision to obtain a fixed-point normalized attention parameter matrix that meets the precision requirements.

[0140] Exemplarily, the floating point normalized attention parameter matrix is ​​a matrix W containing 32-bit floating point parameters float For example, if the preset quantization precision is 8 bits, the matrix W float The 32-bit floating-point parameters in are quantized to 8-bit fixed-point (int8) parameters. That is, the fixed-point normalized attention parameter matrix can be a matrix W containing 8-bit fixed-point parameters. int .

[0141] Of course, the preset quantization precision can also be 16, in which case the matrix W float The 32-bit floating-point parameters in are quantized into 16-bit fixed-point (int8) parameters, which is not specifically limited in the embodiments of the present disclosure.

[0142] In one implementation, during the prefill phase, the floating-point attention parameter matrix needs to be Figure 1The masking layer shown is superimposed with a mask matrix to become a lower triangular matrix, and then normalized. At this time, the first element of the first row is 1, and the rest are 0. From top to bottom, the value of each row is getting smaller and smaller, and the value distribution difference is large. Therefore, the floating-point normalized attention parameter matrix has a large loss when quantized. In order to reduce this loss, the quantization accuracy used in the quantization process of the floating-point normalized attention parameter matrix can be improved. In other words, the preset quantization accuracy corresponding to the fixed-point normalized attention parameter matrix is ​​higher than the quantization accuracy corresponding to the fixed-point value vector feature matrix.

[0143] The disclosed embodiment quantizes each parameter in the floating-point normalized attention parameter matrix based on the global quantization parameter and the preset quantization accuracy to obtain a fixed-point normalized attention parameter matrix, so as to facilitate the subsequent fixed-point calculation of the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and the historical fixed-point value vector feature matrix.

[0144] like Fig.10 As shown in the above Figure 5 Based on the illustrated embodiment, step 442 may include the following steps:

[0145] Step 4421, determine the floating-point key vector feature matrix corresponding to the current information to be inferred.

[0146] In this step, the floating-point key vector feature matrix is ​​the matrix K obtained by the neural network model performing a linear transformation on the current information to be inferred.

[0147] Step 4422, quantize the floating-point key vector feature matrix to obtain a fixed-point key vector feature matrix.

[0148] In this step, each floating-point parameter in the floating-point key vector feature matrix may be quantized based on the quantization parameter corresponding to the floating-point key vector feature matrix. The quantization parameter corresponding to the floating-point key vector feature matrix may be dynamically calculated during the inference process.

[0149] In one implementation, the quantization parameters corresponding to each minimum input unit may include at least a scaling ratio corresponding to each minimum input unit. In this way, the floating-point parameters corresponding to each minimum input unit in the floating-point key vector feature matrix may be mapped to fixed-point parameters of a specified length based on the scaling ratio corresponding to each minimum input unit.

[0150] The disclosed embodiment quantifies each parameter in the floating-point key vector feature matrix to obtain a fixed-point key vector feature matrix, which facilitates the subsequent fixed-point calculation of the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and the historical fixed-point value vector feature matrix. As a result, the amount of floating-point calculations in the reasoning process can be reduced, thereby reducing the computing power resource requirements of the terminal device for answering text determination. At the same time, since the amount of floating-point calculations in the reasoning process is reduced, the reasoning efficiency of the large model is also improved, thereby improving the efficiency of executing the dialogue answering operation.

[0151] like Fig.11 As shown in the above Figure 5 Based on the illustrated embodiment, step 441 may include the following steps:

[0152] Step 4411, determine the floating-point query vector feature matrix corresponding to the current information to be inferred.

[0153] In this step, if Figure 2 In each round of reasoning shown, the floating-point query vector feature matrix is ​​the matrix Q obtained by the neural network model performing a linear transformation on the current information to be reasoned.

[0154] Step 4412, quantize the floating-point query vector feature matrix to obtain a fixed-point query vector feature matrix.

[0155] In this step, the floating-point parameters corresponding to each minimum input unit in the floating-point query vector feature matrix can be quantized based on the quantization parameters corresponding to each token (ie, the minimum input unit). The quantization parameters corresponding to each minimum input unit can be dynamically calculated during the inference process.

[0156] The quantization method of the floating-point query vector feature matrix is ​​similar to the aforementioned floating-point key vector feature matrix, which can be referred to in the aforementioned step 4422 and will not be repeated here.

[0157] In the embodiment of the present disclosure, the characteristic of the quantized floating-point query vector feature matrix is ​​that the floating-point parameters corresponding to each token in the floating-point query vector feature matrix are quantized respectively. Its beneficial effects are similar to those of the embodiment corresponding to the aforementioned steps 4421-4422, and will not be repeated here.

[0158] like Fig.12 As shown in the above Fig.10 Based on the illustrated embodiment, step 4422 may include the following steps:

[0159] Step 4a, based on the numerical range of the parameter corresponding to each minimum input unit in the floating-point key vector feature matrix, determine the quantization parameter corresponding to each minimum input unit in the floating-point key vector feature matrix.

[0160] In this step, the numerical range of the parameter corresponding to each minimum input unit in the floating-point key vector feature matrix is ​​determined, and the quantization parameter corresponding to each minimum input unit is determined according to the numerical range. In this way, the floating-point parameter corresponding to each minimum input unit in the floating-point key vector feature matrix can be mapped to a fixed-point parameter of a specified length based on the quantization parameter corresponding to each minimum input unit.

[0161] Exemplarily, a quantization parameter including a scaling ratio may be determined according to a numerical range of the parameter corresponding to each minimum input unit, and the floating-point parameter corresponding to each minimum input unit in the floating-point key vector feature matrix may be scaled to a fixed-point parameter between 0 and 1.

[0162] Step 4b, based on the quantization parameters corresponding to each minimum input unit in the floating-point key vector feature matrix, quantize the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix to obtain a fixed-point key vector feature matrix.

[0163] In this step, the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix can be quantized based on the quantization parameters corresponding to each minimum input unit in the floating-point key vector feature matrix, so as to convert the floating-point parameters corresponding to each minimum input unit into fixed-point parameters, and obtain a fixed-point key vector feature matrix corresponding to the floating-point key vector feature matrix.

[0164] The quantization method of the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix is ​​similar to the quantization method of the parameters corresponding to each column feature of the aforementioned floating-point value vector feature matrix, which can be referred to in the aforementioned step 43 and will not be repeated here.

[0165] In the disclosed embodiment, the characteristic of the quantized floating-point key vector feature matrix is ​​that the floating-point parameters corresponding to each token in the floating-point key vector feature matrix are quantized respectively. That is to say, the floating-point parameters corresponding to each row in the floating-point key vector feature matrix are quantized respectively. In this way, each row in the fixed-point key vector feature matrix can correspond to the same quantization parameter. Similarly, each row in the historical fixed-point key vector feature matrix also corresponds to the same quantization parameter, so each row in the fixed-point key vector splicing feature matrix corresponds to the same quantization parameter. In this way, each column in the transposed matrix of the historical fixed-point key vector feature matrix corresponds to the same quantization parameter, which conforms to the calculation method of general matrix multiplication, and the elements in the fixed-point query vector feature matrix and the transposed matrix of the fixed-point key vector splicing feature matrix are directly accumulated after corresponding multiplication. Therefore, the transposed matrix of the fixed-point query vector feature matrix and the fixed-point key vector concatenation feature matrix can be used for matrix multiplication, without the need to dequantize the fixed-point key vector feature matrix, which can reduce the amount of computational effort in model reasoning, that is, reduce the demand for computing power resources of the terminal device for reasoning operations, improve the computational efficiency of large model reasoning operations, and thus improve the efficiency of executing dialogue response operations.

[0166] like Fig.13 As shown in the above Fig.11 Based on the illustrated embodiment, step 4412 may include the following steps:

[0167] Step 5a: Determine the quantization parameter corresponding to each minimum input unit in the floating-point query vector feature matrix based on the value range of the parameter corresponding to each minimum input unit in the floating-point query vector feature matrix.

[0168] In this step, the numerical range of the parameter corresponding to each minimum input unit in the floating-point query vector feature matrix is ​​determined, and the quantization parameter corresponding to each minimum input unit is determined according to the numerical range. In this way, the floating-point parameter corresponding to each minimum input unit in the floating-point query vector feature matrix can be mapped to a fixed-point parameter of a specified length based on the quantization parameter corresponding to each minimum input unit.

[0169] The method for determining the quantization parameter corresponding to each minimum input unit in the floating-point query vector feature matrix is ​​similar to that of the aforementioned floating-point key vector feature matrix, which can be referred to in the aforementioned step 4a and will not be repeated here.

[0170] Step 5b, based on the quantization parameters corresponding to each minimum input unit in the floating-point query vector feature matrix, quantize the parameters corresponding to each minimum input unit in the floating-point query vector feature matrix to obtain a fixed-point query vector feature matrix.

[0171] In this step, the parameters corresponding to each minimum input unit in the floating-point query vector feature matrix can be quantized based on the quantization parameters corresponding to each minimum input unit in the floating-point query vector feature matrix, so as to convert the floating-point parameters corresponding to each minimum input unit into fixed-point parameters, and obtain a fixed-point query vector feature matrix corresponding to the floating-point query vector feature matrix.

[0172] The quantization method of the parameters corresponding to each minimum input unit in the floating-point query vector feature matrix is ​​similar to that of the aforementioned floating-point key vector feature matrix, which can be referred to in the aforementioned step 4b and will not be repeated here.

[0173] In the embodiment of the present disclosure, the characteristic of the quantized floating-point query vector feature matrix is ​​that the floating-point parameters corresponding to each token in the floating-point query vector feature matrix are quantized respectively. Its beneficial effects are similar to those of the embodiment corresponding to the aforementioned steps 4a-4b, and will not be repeated here.

[0174] Fig.14 is a schematic diagram of a matrix multiplication layer provided by an exemplary embodiment of the present disclosure.

[0175] like Fig.14 As shown, the first matrix multiplication calculation is performed in the first matrix multiplication layer. Specifically, the fixed-point query vector feature matrix Q int The matrix obtained by concatenating the fixed-point key vector feature matrix and the historical fixed-point key vector feature matrix Perform matrix multiplication on the transposed matrix to obtain the fixed-point attention parameter matrix.

[0176] Among them, the historical fixed-point key vector feature matrix can be read from the KV Cache. The fixed-point query vector feature matrix is ​​quantized by the parameters corresponding to each minimum input unit of the floating-point query vector feature matrix. The fixed-point key vector feature matrix is ​​quantized by the parameters corresponding to each minimum input unit of the floating-point key vector feature matrix. Exemplarily, the quantization accuracy of the fixed-point query vector feature matrix and the fixed-point key vector feature matrix can be int8.

[0177] The second matrix multiplication calculation is performed in the second matrix multiplication layer. Specifically, the matrix obtained by concatenating the fixed-point normalized attention parameter matrix with the fixed-point value vector feature matrix and the historical fixed-point value vector feature matrix is Do matrix multiplication.

[0178] Among them, the historical fixed-point value vector feature matrix can be read from the KV Cache. The fixed-point value vector feature matrix is ​​obtained by quantizing the parameters corresponding to each column feature of the floating-point value vector feature matrix. The fixed-point normalized attention parameter matrix is ​​obtained by dequantizing, normalizing and quantizing the fixed-point attention parameter matrix in sequence. Exemplarily, the quantization accuracy of the fixed-point value vector feature matrix can be int8, and the quantization accuracy of the fixed-point normalized attention parameter matrix can be 16int.

[0179] like Fig.15 As shown in the above Figure 5 Based on the illustrated embodiment, step 443 may include the following steps:

[0180] Step 4431, concatenate the fixed-point key vector feature matrix and at least one historical fixed-point key vector feature matrix to obtain a fixed-point key vector concatenation feature matrix.

[0181] In this step, the Attention structure can access the key-value cache space, read at least one historical fixed-point key vector feature matrix in the cache, and concatenate the fixed-point key vector feature matrix with at least one historical fixed-point key vector feature matrix to obtain a fixed-point key vector concatenation feature matrix corresponding to the current information to be inferred and the historical information to be inferred.

[0182] The concatenation method of the fixed-point key vector concatenation feature matrix is ​​similar to the concatenation method of the fixed-point value vector concatenation feature matrix mentioned above, and can be referred to the above step 1a, which will not be repeated here.

[0183] Step 4432, obtain the transposed matrix corresponding to the fixed-point key vector concatenated feature matrix.

[0184] In this step, the fixed-point key vector is concatenated with the feature matrix Perform transpose calculation to obtain the transposed matrix

[0185] Step 4433, perform matrix multiplication on the transposed matrix corresponding to the fixed-point query vector feature matrix and the fixed-point key vector concatenated feature matrix to obtain a fixed-point attention parameter matrix.

[0186] In this step, matrix multiplication is performed on the transposed matrix corresponding to the fixed-point query vector feature matrix and the fixed-point key vector concatenation feature matrix, such as Fig.14 In the second matrix multiplication layer

[0187] Fig.16 A schematic diagram of matrix multiplication is provided for an exemplary embodiment of the present disclosure.

[0188] like Fig.16As shown, the fixed-point query vector feature matrix Q int The parameters of each row in the transposed matrix The corresponding parameters of each column in are multiplied and accumulated to obtain the fixed-point attention parameter matrix.

[0189] In the disclosed embodiment, the fixed-point key vector splicing feature matrix is ​​characterized in that each parameter on the same row corresponds to the same quantization parameter. Therefore, each parameter on the same column of the transposed matrix of the fixed-point key vector splicing feature matrix corresponds to the same quantization parameter. This characteristic makes it consistent with the calculation method of general matrix multiplication. Therefore, the transposed matrix corresponding to the fixed-point query vector feature matrix and the fixed-point key vector splicing feature matrix can be directly multiplied without additional inverse quantization calculation. For example, the transposed matrix In the same column, the parameters correspond to the same scaling ratio. Therefore, the fixed-point query vector feature matrix Q int and the transposed matrix Matrix multiplication can be performed directly. This can reduce the amount of floating-point calculations in the reasoning process, thereby reducing the computing power resource requirements for answering text determination, improving the efficiency of large model reasoning calculations, and thus improving the efficiency of executing dialogue answering operations.

[0190] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0191] Exemplary Devices

[0192] Fig.17 An exemplary embodiment of the present disclosure provides a device for determining a response text based on a conversation text.

[0193] like Fig.17 As shown, the device 170 for determining a response text based on a conversation text includes an input module 171 , a linearization module 172 , a quantization module 173 , an output module 174 and a response module 175 .

[0194] An input module 171, used to determine the current information to be inferred based on the user's conversation text and the current inference round of the neural network model;

[0195] A linearization module 172, used to determine a floating-point value vector feature matrix corresponding to the current information to be inferred, wherein the floating-point value vector feature matrix includes multiple columns of features;

[0196] A quantization module 173, configured to quantize a floating-point value vector feature matrix based on a quantization parameter corresponding to each column of features in a predetermined plurality of columns of features, to obtain a fixed-point value vector feature matrix corresponding to the current information to be inferred;

[0197] An output module 174, configured to obtain a current reasoning result corresponding to the current information to be inferred based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space;

[0198] The response module 175 is used to determine the response text corresponding to the dialogue text according to the current reasoning result.

[0199] Fig.18 A device for determining a response text based on a conversation text is provided as another exemplary embodiment of the present disclosure.

[0200] like Fig.18 As shown, the output module 174 includes a quantization unit 1741, an attention calculation unit 1742 and an inference result output unit 1743.

[0201] The quantization unit 1741 is used to determine the fixed-point query vector feature matrix corresponding to the floating-point query vector feature matrix corresponding to the current information to be inferred; determine the fixed-point key vector feature matrix corresponding to the floating-point key vector feature matrix corresponding to the current information to be inferred.

[0202] The attention calculation unit 1742 is used to obtain a fixed-point attention parameter matrix corresponding to the dialogue text based on a fixed-point query vector feature matrix, a fixed-point key vector feature matrix, and at least one historical fixed-point key vector feature matrix corresponding to historical information to be inferred that is previously cached in the key-value cache space.

[0203] The inference result output unit 1743 is used to obtain the current inference result based on the fixed-point attention parameter matrix, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix.

[0204] In some embodiments, the inference result output unit 1743 is used to dequantize the fixed-point attention parameter matrix to obtain the floating-point attention parameter matrix corresponding to the dialogue text; normalize the floating-point attention parameter matrix to obtain the floating-point normalized attention parameter matrix; quantize the floating-point normalized attention parameter matrix to obtain the fixed-point normalized attention parameter matrix; and obtain the current inference result based on the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix.

[0205] In some embodiments, the inference result output unit 1743 is used to splice the fixed-point value vector feature matrix and at least one historical fixed-point value vector feature matrix to obtain a fixed-point value vector splicing feature matrix; perform matrix multiplication calculation on the fixed-point normalized attention parameter matrix and the fixed-point value vector splicing feature matrix to obtain the current inference result.

[0206] In some embodiments, the inference result output unit 1743 is used to determine the global quantization parameter shared by each parameter in the floating-point normalized attention parameter matrix based on the numerical range of each attention value in the floating-point normalized attention parameter matrix; based on the global quantization parameter, quantize the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix.

[0207] In some embodiments, the inference result output unit 1743 is used to quantize the floating-point normalized attention parameter matrix based on the global quantization parameter with a preset quantization precision to obtain a fixed-point normalized attention parameter matrix corresponding to the preset quantization precision, and the preset quantization precision is higher than the quantization precision corresponding to the fixed-point value vector feature matrix.

[0208] In some embodiments, the quantization unit 1741 is used to determine the floating-point key vector feature matrix corresponding to the current information to be inferred; quantize the floating-point key vector feature matrix to obtain a fixed-point key vector feature matrix.

[0209] In some embodiments, the fixed-point vector feature matrix determining unit is used to determine the floating-point query vector feature matrix corresponding to the current information to be inferred; and quantize the floating-point query vector feature matrix to obtain a fixed-point query vector feature matrix.

[0210] In some embodiments, the quantization unit 1741 is used to determine the quantization parameters corresponding to each minimum input unit in the floating-point key vector feature matrix based on the numerical range of the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix; based on the quantization parameters corresponding to each minimum input unit in the floating-point key vector feature matrix, the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix are quantized to obtain a fixed-point key vector feature matrix.

[0211] In some embodiments, the quantization unit 1741 is used to determine the quantization parameter corresponding to each minimum input unit in the floating-point query vector feature matrix based on the numerical range of the parameter corresponding to each minimum input unit in the floating-point query vector feature matrix; based on the quantization parameter corresponding to each minimum input unit in the floating-point query vector feature matrix, the parameter corresponding to each minimum input unit in the floating-point query vector feature matrix is ​​quantized to obtain a fixed-point query vector feature matrix.

[0212] In some embodiments, the attention calculation unit 1742 is used to splice a fixed-point key vector feature matrix and at least one historical fixed-point key vector feature matrix to obtain a fixed-point key vector splicing feature matrix; obtain the transposed matrix corresponding to the fixed-point key vector splicing feature matrix; perform matrix multiplication calculation on the transposed matrix corresponding to the fixed-point query vector feature matrix and the fixed-point key vector splicing feature matrix to obtain a fixed-point attention parameter matrix.

[0213] Exemplary Electronic Devices

[0214] Fig.19 The structural diagram of an electronic device 190 provided for an exemplary embodiment of the present disclosure includes at least one processor 191 and a memory 192 .

[0215] The processor 191 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 190 to perform desired functions.

[0216] The memory 192 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 191 may execute one or more computer program instructions to implement the operating mode switching method and / or other desired functions of the system-on-chip of each embodiment of the present disclosure above.

[0217] In one example, the electronic device 190 may further include: an input device 193 and an output device 194 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0218] The input device 193 may also include, for example, a keyboard, a mouse, etc.

[0219] The output device 194 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0220] Of course, to simplify, Fig.19 Only some of the components related to the present disclosure in the electronic device 190 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 190 may also include any other appropriate components.

[0221] In one embodiment, the processor of the electronic device 190 may be used to execute the steps of the method for determining a response text based on a conversation text in various embodiments of the present disclosure described in the above “Exemplary Method” section.

[0222] Exemplary computer program products and computer-readable storage media

[0223] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the operating mode switching method of the system on chip of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0224] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0225] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps in the operating mode switching method of the on-chip system of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.

[0226] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0227] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.

[0228] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the present claims and their equivalents, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for determining a response text based on a conversation text, comprising: Determine the current information to be inferred based on the user's conversation text and the current inference round of the neural network model; Determine a floating-point value vector feature matrix corresponding to the current information to be inferred, wherein the floating-point value vector feature matrix includes multiple columns of features; quantizing the floating-point value vector feature matrix based on the predetermined quantization parameters corresponding to each column of the plurality of columns of features, to obtain a fixed-point value vector feature matrix corresponding to the current information to be inferred; Based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space, obtaining a current reasoning result corresponding to the current information to be inferred; Determine a response text corresponding to the dialogue text according to the current inference result.

2. The method for determining a response text based on a conversation text according to claim 1, wherein: The obtaining, based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space, a current reasoning result corresponding to the current information to be inferred, comprises: Determine a fixed-point query vector feature matrix corresponding to the floating-point query vector feature matrix corresponding to the current information to be inferred; Determine a fixed-point key vector feature matrix corresponding to the floating-point key vector feature matrix corresponding to the current information to be inferred; Obtaining a fixed-point attention parameter matrix corresponding to the dialogue text according to the fixed-point query vector feature matrix, the fixed-point key vector feature matrix, and at least one historical fixed-point key vector feature matrix corresponding to the historical information to be inferred that is previously cached in the key-value cache space; The current inference result is obtained based on the fixed-point attention parameter matrix, the fixed-point value vector feature matrix, and the at least one historical fixed-point value vector feature matrix.

3. The method for determining a response text based on a conversation text according to claim 2, wherein: The obtaining the current inference result based on the fixed-point attention parameter matrix, the fixed-point value vector feature matrix, and the at least one historical fixed-point value vector feature matrix includes: Dequantizing the fixed-point attention parameter matrix to obtain a floating-point attention parameter matrix corresponding to the dialogue text; Normalizing the floating-point attention parameter matrix to obtain a floating-point normalized attention parameter matrix; quantizing the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix; The current inference result is obtained based on the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and the at least one historical fixed-point value vector feature matrix.

4. The method for determining a response text based on a conversation text according to claim 3, wherein: The obtaining the current inference result based on the fixed-point normalized attention parameter matrix, the fixed-point value vector feature matrix, and the at least one historical fixed-point value vector feature matrix includes: splicing the fixed-point valued vector feature matrix and the at least one historical fixed-point valued vector feature matrix to obtain a fixed-point valued vector splicing feature matrix; Perform matrix multiplication on the fixed-point normalized attention parameter matrix and the fixed-point value vector concatenation feature matrix to obtain the current inference result.

5. The method for determining a response text based on a conversation text according to claim 3, wherein: The quantizing the floating-point normalized attention parameter matrix to obtain a fixed-point normalized attention parameter matrix includes: Determining a global quantization parameter shared by each parameter in the floating-point normalized attention parameter matrix based on a numerical range of each attention value in the floating-point normalized attention parameter matrix; Based on the global quantization parameter, the floating-point normalized attention parameter matrix is ​​quantized to obtain the fixed-point normalized attention parameter matrix.

6. The method for determining a response text based on a conversation text according to claim 5, wherein: The step of quantizing the floating-point normalized attention parameter matrix based on the global quantization parameter to obtain the fixed-point normalized attention parameter matrix includes: Based on the global quantization parameter, the floating-point normalized attention parameter matrix is ​​quantized with a preset quantization precision to obtain the fixed-point normalized attention parameter matrix corresponding to the preset quantization precision, and the preset quantization precision is higher than the quantization precision corresponding to the fixed-point value vector feature matrix.

7. The method for determining a response text based on a conversation text according to claim 2, wherein: The step of determining a fixed-point key vector feature matrix corresponding to the floating-point key vector feature matrix corresponding to the current information to be inferred includes: Determine a floating-point key vector feature matrix corresponding to the current information to be inferred; The floating-point key vector feature matrix is ​​quantized to obtain the fixed-point key vector feature matrix.

8. The method for determining a response text based on a conversation text according to claim 2, wherein: The step of determining a fixed-point query vector feature matrix corresponding to the floating-point query vector feature matrix corresponding to the current information to be inferred includes: Determine a floating-point query vector feature matrix corresponding to the current information to be inferred; The floating-point query vector feature matrix is ​​quantized to obtain the fixed-point query vector feature matrix.

9. The method for determining a response text based on a conversation text according to claim 7, wherein: The quantizing the floating-point key vector feature matrix to obtain the fixed-point key vector feature matrix includes: Determine the quantization parameter corresponding to each minimum input unit in the floating-point key vector feature matrix based on the numerical range of the parameter corresponding to each minimum input unit in the floating-point key vector feature matrix; Based on the quantization parameters corresponding to each minimum input unit in the floating-point key vector feature matrix, the parameters corresponding to each minimum input unit in the floating-point key vector feature matrix are quantized to obtain the fixed-point key vector feature matrix.

10. The method for determining a response text based on a conversation text according to claim 8, wherein: The quantizing the floating-point query vector feature matrix to obtain the fixed-point query vector feature matrix includes: Determine the quantization parameter corresponding to each minimum input unit in the floating-point query vector feature matrix based on the numerical range of the parameter corresponding to each minimum input unit in the floating-point query vector feature matrix; Based on the quantization parameters corresponding to each minimum input unit in the floating-point query vector feature matrix, the parameters corresponding to each minimum input unit in the floating-point query vector feature matrix are quantized to obtain the fixed-point query vector feature matrix.

11. The method for determining a response text based on a conversation text according to claim 2, wherein: According to the fixed-point query vector feature matrix, the fixed-point key vector feature matrix, and at least one historical fixed-point key vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space, a fixed-point attention parameter matrix corresponding to the dialogue text is obtained, including: Splicing the fixed-point key vector feature matrix and the at least one historical fixed-point key vector feature matrix to obtain a fixed-point key vector splicing feature matrix; Obtaining a transposed matrix corresponding to the fixed-point key vector concatenation feature matrix; A matrix multiplication is performed on the transposed matrix corresponding to the fixed-point query vector feature matrix and the fixed-point key vector concatenated feature matrix to obtain the fixed-point attention parameter matrix.

12. A device for determining a response text based on a conversation text, comprising: An input module, configured to determine current information to be inferred based on the user's conversation text and the current inference round of the neural network model; A linearization module is configured to determine a floating-point value vector feature matrix corresponding to the current information to be inferred, wherein the floating-point value vector feature matrix includes multiple columns of features; A quantization module, configured to quantize the floating-point value vector feature matrix based on a predetermined quantization parameter corresponding to each column of the plurality of columns of features, to obtain a fixed-point value vector feature matrix corresponding to the current information to be inferred; an output module configured to obtain a current reasoning result corresponding to the current information to be inferred based on the current information to be inferred, the fixed-point value vector feature matrix, and at least one historical fixed-point value vector feature matrix corresponding to the historical information to be inferred previously cached in the key-value cache space; The response module is configured to determine the response text corresponding to the dialogue text according to the current reasoning result.

13. An electronic device comprising: One or more processors, and a memory; the memory stores computer instructions; the computer instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 11.

14. A computer-readable storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 11.

Images