Text processing method and device, storage medium and computer equipment

Abstract

The invention provides a text processing method and device, a storage medium and computer equipment, and the method comprises the steps: determining a reasoning mode and reasoning parameters of a neural network model according to the text length and semantic complexity of an input text; determining an initial hidden state of the neural network model and a key value pair multiplexing result according to the historical context of the input text and the key value pair cache fragment of the historical context; determining candidate output representation of the input text in an initial hidden state by utilizing a neural network model according to the reasoning mode, the key value pair multiplexing result and the reasoning parameters; generating target output representations matched with the input text according to the sampling temperatures matched with the importance of the candidate output representations, and returning to the step of determining the initial hidden state of the neural network model and the key value pair multiplexing result until each target output representation matched with the input text is generated, and obtaining an answer text corresponding to the input text.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a text processing method, apparatus, storage medium, and computer device. Background Art

[0002] With the advancement of artificial intelligence (AI), large-scale language models (LLMs) are increasingly emerging. LLMs not only play a vital role in natural language processing tasks but also demonstrate outstanding performance in specific fields such as finance, healthcare, and education, leading to their widespread application. However, current LLMs for text processing suffer from significant drawbacks, such as wasted inference resources and prolonged inference times. Therefore, reducing resource waste and accelerating inference have become a worthy technical challenge. Summary of the Invention

[0003] In view of this, the present application provides a text processing method, apparatus, storage medium and computer equipment, aiming to improve model reasoning speed and reduce computing costs.

[0004] Specifically, this application is implemented through the following technical solutions:

[0005] In a first aspect, an embodiment of the present disclosure provides a text processing method, comprising:

[0006] Determine the reasoning mode and reasoning parameters of the neural network model based on the text length and semantic complexity of the input text;

[0007] Determining an initial hidden state of the neural network model and a key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context;

[0008] Determining, using the neural network model, candidate output representations of the input text in the initial hidden state according to the inference pattern, the key-value pair reuse result, and the inference parameter;

[0009] According to the sampling temperature that matches the importance of the candidate output representation, a target output representation that matches the input text is generated, and the step of determining the initial hidden state of the neural network model and the key-value pair multiplexing result is returned until each target output representation that matches the input text is generated, and an answer text corresponding to the input text is obtained.

[0010] In one possible implementation, determining the reasoning mode and reasoning parameters of the neural network model based on the text length and semantic complexity of the input text includes:

[0011] Selecting the inference mode from preset modes according to the text length and the length threshold of the input text; the preset modes include full network layer inference and partial network layer inference;

[0012] According to the semantic complexity, a weight screening parameter and a decoding step size for the input representation are determined, and the weight screening parameter and the decoding step size are used as the inference parameters.

[0013] In one possible implementation, determining, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter, includes:

[0014] Determining, using the neural network model, an initial attention weight for each input representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the decoding step size;

[0015] Sort the initial attention weights in descending order to obtain a sorting order;

[0016] The initial attention weight whose sorting order is less than or equal to the weight screening parameter is used as the target attention weight of the input representation;

[0017] Determine the candidate output representation based on the target attention weight and the input representation.

[0018] In one possible implementation, determining, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter, includes:

[0019] Determining the inference time of the neural network model under different inference paths;

[0020] Determining computing resources to allocate to each input representation based on the importance of each input representation of the input text and the importance of the generated target output representation;

[0021] The neural network model is used to use the computing resources allocated to each input representation, and according to the reasoning path with the shortest reasoning time, the candidate output representation of the input text is determined in the initial hidden state according to the reasoning mode, the key-value pair reuse result and the reasoning parameters.

[0022] In a possible implementation, before returning the step of determining the initial hidden state of the neural network model and the key-value pair reuse result, the method further includes:

[0023] Determining the time and attention computational proportion for generating the target output representation;

[0024] Adjusting a key-value pair reuse rate according to the time; the key-value pair reuse rate is used to adjust the proportion of reused key-value pairs in a key-value pair reuse result used in the next calculation of the candidate output representation;

[0025] The inference parameters are adjusted according to the semantic complexity corresponding to the generated target output representation, the semantic complexity of the input text, and the attention calculation ratio.

[0026] In one possible implementation, determining the initial hidden state of the neural network model and the key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context includes:

[0027] When the text similarity between the historical context and the input text is greater than a set similarity, determining a matching degree between the input text and the key-value pair cache segment based on the key-value pair cache segment of the historical context and a feature vector of the input text;

[0028] The key-value pair reuse result is determined according to the matching degree, and the initial hidden state is determined according to the encoding result of the historical context.

[0029] In one possible implementation, the method further includes:

[0030] If the historical context does not exist or the text similarity is not greater than the set similarity, the key-value pair reuse result is determined to be empty, and the initial hidden state is determined based on the hidden states corresponding to the stored target texts and the input text; wherein the target text includes text whose input frequency is higher than the set frequency during the model training process.

[0031] In a second aspect, an embodiment of the present disclosure further provides a text processing device, comprising:

[0032] A first determination module is used to determine the reasoning mode and reasoning parameters of the neural network model according to the text length and semantic complexity of the input text;

[0033] A second determination module is configured to determine an initial hidden state of the neural network model and a key-value pair reuse result based on a historical context of the input text and a key-value pair cache segment of the historical context;

[0034] a third determining module, configured to determine, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter;

[0035] A generation module is used to generate a target output representation that matches the input text according to a sampling temperature that matches the importance of the candidate output representation, and return to the step of determining the initial hidden state of the neural network model and the key-value pair multiplexing result until each target output representation that matches the input text is generated to obtain an answer text corresponding to the input text.

[0036] In a third aspect, an optional implementation of the present disclosure further provides a computer-readable storage medium, comprising a computer program, which, when executed, implements the above-mentioned first aspect, or the steps in any possible implementation of the first aspect.

[0037] In a fourth aspect, an optional implementation of the present disclosure further provides a computer device, a processor, and a memory, wherein the memory stores machine-readable instructions executable by the processor, and the processor is used to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0038] The text processing method provided by the embodiments of the present disclosure first determines the most suitable inference mode and inference parameters for the input text based on the text length and semantic complexity, thereby dynamically optimizing the optimal calculation process. This avoids the waste of inference resources by using a neural network model to infer all input texts according to a fixed calculation process using all parameters. By determining an initial hidden state suitable for the input text for the neural network model and starting inference from the initial hidden state, the problem of low inference efficiency caused by the model inferring from a zero state can be avoided. By reusing key-value pair cache fragments, redundant calculations during the inference process can be effectively reduced, thereby improving inference efficiency. By determining the sampling temperature based on the importance of the candidate output and using the sampling temperature to generate the target output representation, different sampling temperatures can be set for different candidate output representations, thereby flexibly controlling the diversity of the target output representation generation. In general, by selecting different inference modes, inference parameters, initial hidden states, key-value pair reuse results, sampling temperatures, and other information used in model inference for different input texts, and controlling the neural network model to perform inference according to this information, the present application can effectively prioritize inference paths, improve inference speed, and reduce resource waste.

[0039] For a description of the effects of the above-mentioned text processing apparatus, computer equipment, and computer program product, please refer to the description of the above-mentioned text processing method, which will not be repeated here.

[0040] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flowchart of a text processing method shown in an exemplary embodiment of the present application;

[0042] Figure 2 1 is a schematic diagram of an initial hidden state determination process shown in an exemplary embodiment of the present application;

[0043] Figure 3 This is a flowchart of a specific implementation process of a text processing method shown in an exemplary embodiment of the present application;

[0044] Figure 4 This is a hardware structure diagram of a computer device where a text processing device 500 is located, as shown in an exemplary embodiment of the present application;

[0045] Figure 5 It is a structural diagram of a text processing device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0046] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0047] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0049] In this document, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0050] Research has found that when using LLM for text processing, the traditional inference method is to output the text to the large model directly after it is deployed, and then the large model performs inference based on its own structure. This method results in a fixed computational path and execution of the same calculation process for all text, failing to optimize inference for different text characteristics, resulting in wasted resources and inference performance bottlenecks. Furthermore, current inference methods often use a decoding strategy that initializes from zero, ignoring information that has already appeared during the inference process, which affects inference efficiency.

[0051] Based on the above research, the present disclosure provides a text processing method, device, storage medium and computer equipment, which are intended to optimize input analysis, initial state setting, intermediate calculation path, decoding strategy and computing resource allocation to improve reasoning speed and reduce computing cost. First, based on the text length and semantic complexity, the most matching reasoning mode and reasoning parameters can be determined for the input text, thereby dynamically optimizing the optimal calculation process, rather than using the neural network model to perform reasoning using all parameters according to a fixed calculation process for all input texts, thereby avoiding the waste of reasoning resources. By determining the initial hidden state suitable for the input text for the neural network model and starting reasoning from the initial hidden state, the problem of low reasoning efficiency caused by the model reasoning from zero state can be avoided. By reusing key-value pair cache fragments, redundant calculations in the reasoning process can be effectively reduced, thereby improving reasoning efficiency. By determining the sampling temperature according to the importance of the candidate output and using the temperature to generate the target output representation, different sampling temperatures can be set for different candidate output representations, thereby flexibly controlling the diversity of target output representation generation. In general, this application can effectively prioritize reasoning paths, improve reasoning speed and reduce resource waste by selecting different reasoning modes, reasoning parameters, initial hidden states, key-value pair reuse results, temperature and other information used in model reasoning for different input texts, and controlling the neural network model to reason according to this information.

[0052] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the present disclosure for the above problems below are all contributions made by the inventors to the present disclosure during the disclosure process.

[0053] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0054] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0055] To facilitate understanding of this embodiment, a text processing method disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the text processing method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capabilities, where the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a personal digital assistant (PDA), a handheld device, a computer device, etc.; in some possible implementation methods, the text processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0056] The text processing method provided by the embodiment of the present disclosure is described below by taking the execution subject as a server as an example.

[0057] like Figure 1 FIG. 1 is a flowchart of a text processing method provided by an embodiment of the present disclosure, which may include the following steps:

[0058] S101: Determine the reasoning mode and reasoning parameters of the neural network model according to the text length and semantic complexity of the input text.

[0059] Here, the input text is the text for model inference. This text can be of any length and in any language. The text length can be determined by the number of input representations (i.e., input tokens) corresponding to the input text. For example, if the input text X contains N input tokens, then the length of the input text is L(X) = N.

[0060] Semantic complexity is used to characterize the semantic perplexity of an input text. Specifically, the perplexity (PPL) calculation module can be used to calculate the perplexity of the input text.

[0061] The neural network model can be a pre-trained large language model. The inference mode indicates the computational model used by the neural network model when performing textual inference on the input text. The inference parameters indicate the parameters used by the neural network model when performing textual inference on the input text. For example, these parameters may include the number of transformer layers, decoding step size, weight filtering parameters for attention weights, and the amount of attention computation required.

[0062] In specific implementation, after receiving the inference request, the server can obtain the input text from the inference request, perform feature analysis on the input text, extract and quantify multiple key indicators of the input text, and optimize the inference path. Among them, the key indicators may include the text length L(X), semantic complexity PPL(X) of the input text, and the feature vector described later. After obtaining the key indicators, the inference mode and inference parameters required for the neural network model to be used for the input text can be determined based on the text length and semantic complexity. For example, the inference mode currently required can be determined based on the correspondence between the text length and different inference modes; and the various inference parameters required to be used can be determined based on the range of semantic complexity.

[0063] In one embodiment, the above S101 can be implemented according to the following steps:

[0064] S101-1: Select an inference mode from preset modes based on the text length and length threshold of the input text; the preset modes include full network layer inference and partial network layer inference.

[0065] Here, full-network-layer inference refers to reasoning using all layers of the neural network layer. This inference mode is also called full-path model inference. Partial-network-layer inference refers to reasoning using only a portion of the neural network layer. This inference mode is also called lightweight inference. For example, it can reduce the number of Transformer layers or use quantized computing to speed up response time.

[0066] The length threshold can be set based on experience and is not specifically limited in the embodiments of the present application. For example, it can be determined based on the capabilities of the neural network model.

[0067] In specific implementation, the text length L(X) of the input text and the text threshold L Threshold Compare. If L(X)<L Threshold , we can determine that the input text is a short text, and then we can use some network layer reasoning modes as the reasoning modes that need to be used to improve the reasoning and response speed; if L(X)≥L Threshold , it can be determined that the input text is a long text, and the full network layer reasoning mode can be used as the reasoning mode to ensure the reasoning accuracy.

[0068] S101-2: Determine weight screening parameters and decoding step size for the input representation according to the semantic complexity, and use the weight screening parameters and decoding step size as inference parameters.

[0069] Here, the weight screening parameter may be a parameter for screening the attention weight of the input text. The decoding step size is the step size required by the mode during the decoding process.

[0070] In specific implementation, the semantic complexity PPL(X) of the input text can be compared with the complexity threshold PPL Threshold Compare. If PPL(X)<PPL Threshold , then the text structure is considered to be relatively simple and the input text belongs to simple text input, so the allocation of computing resources can be reduced, and the weight screening parameters, decoding step size and attention calculation amount that match the simple text input can be used. If PPL(X)≥PPL Threshold , the text structure is considered complex and the input text is considered complex. Therefore, a stronger decoding strategy can be adopted for the input text, increasing computing resources and using weight screening parameters, decoding step size, and attention calculation amount that match complex text input. Among them, the weight screening parameters, decoding step size, and attention calculation amount that match simple text input must be smaller than those that match complex text input.

[0071] S102: Determine the initial hidden state of the neural network model and the key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context.

[0072] Here, the historical context is the processed context text related to the input text, and the key-value cache (KVcache) fragment is the historical key-value pairs generated and cached when processing the historical context, and / or the historical key-value pairs generated and cached when processing the input text.

[0073] The hidden state indicates the internal state maintained by the model when processing each time step. The initial hidden state is the hidden state that the neural network model first uses when processing the input text. The key-value pair reuse result indicates the historical key-value pairs that need to be reused when processing the input text.

[0074] In a specific implementation, when the input text has a historical context, the initial hidden state can be determined based on the hidden states used when processing the historical context, and the historical key-value pairs to be used are selected from the key-value pair cache fragments related to the historical context as the key-value pair reuse result. For example, the initial hidden state can be determined based on the state mean of the hidden states used when processing the historical context.

[0075] In one embodiment, the above step S102 may be implemented as follows:

[0076] S102-1: When the text similarity between the historical context and the input text is greater than a set similarity, determine the matching degree between the input text and the key-value pair cache segment based on the key-value pair cache segment of the historical context and the feature vector of the input text.

[0077] Here, the similarity can be set based on experience and is not specifically limited in the embodiments of the present application.

[0078] In specific implementation, after obtaining the input text, it can be determined whether the input text has a related historical context, that is, whether the input text is a new conversation. If not, the historical context related to the input text can be obtained and the key-value pair cache fragment related to the historical context can be obtained. Wherein, the historical context C = {X1, X2, ... X m}, where X1~X m Represents historical context X1~X m Each input Token, the key-value cache fragment H in the KVcache history memory = {h1, h2, ... h m}, where h1~h m A historical key-value cache representing the feature vectors of each historical context.

[0079] Furthermore, the text similarity between the input text and the historical context can be calculated. The text similarity here can be the similarity between the input text and each historical context separately, or the overall similarity between the input text and each historical context. If the text similarity is greater than the set similarity, the cosine similarity sim(V(X), H) between the key-value cache fragment H of the historical context and the feature vector V(X) of the input text can be calculated, and this cosine similarity can be used as the matching degree between the input text and the key-value cache fragment.

[0080] S102-2: Determine the key-value pair reuse result based on the matching degree, and determine the initial hidden state based on the encoding result of the historical context.

[0081] During specific implementation, the matching degree can be compared with the preset matching degree. When the matching degree is greater than the preset matching degree, it can be determined that the key-value pair cache fragments need to be reused, and based on the current key-value pair reuse rate, the various historical key-value pairs that need to be reused are determined to obtain the key-value pair reuse result. In the case where the matching degree does not preset the matching degree, it can be determined that the key-value pair cache fragments do not need to be reused, and the key-value pair reuse result is determined to be empty. It is understandable that if the key-value pair reuse result determined last time is empty, then when the key-value pair reuse rate is increased next time, it is determined whether to reuse the key-value pair based on the matching degree between the various key-value pairs generated when the target output representation was generated last time and the input text. If so, the key-value pair reuse rate is increased. If not, it is determined that the key-value pair result is still empty and the key-value pair reuse rate is still zero.

[0082] At the same time, in order to improve the reasoning convergence speed, the optimal initial hidden state of the neural network model can be searched in the model decoding stage. Specifically, the historical context of the input text can be encoded using the Transformer encoder to obtain the encoding result h c , h c As the initial hidden state of the neural network model. For example, use the Transformer encoder to encode the historical context C = {X1, X2, ... X m} to encode and get the encoding result h c = TransformerEncoder(C). Using this initial hidden state for model inference can reduce the early convergence time of the model.

[0083] In another embodiment, if there is no historical context or the text similarity is not greater than the set similarity, the key-value pair reuse result is determined to be empty, and the initial hidden state is determined based on the hidden states corresponding to each stored target text and the input text; wherein the target text includes text whose input frequency is higher than the set frequency during the model training process.

[0084] Here, in the process of pre-training the neural network model, this application can also store the hidden states corresponding to some common target texts. Among them, the target texts include texts that are input more frequently than the set frequency during the model training process, such as some common greetings and some frequently searched questions.

[0085] For example, if the input text does not have a historical context or the text similarity between the input text and the historical context is not greater than a set similarity, the key-value pair reuse result can be determined to be empty, that is, it is determined that there is no historical key-value pair cache that can be reused. At this time, it is also possible to determine whether there is a usable hidden state from the hidden state corresponding to the target text. If so, the stored hidden state can be used; if not, it can be determined that the initial hidden state is an all-zero state.

[0086] For example, the initial hidden state can be searched according to the following formula 3:

[0087]

[0088] in, represents the initial hidden state searched, Represents the matching probability, Y represents the target text, X represents the input text, and h0 represents the hidden state stored for the target text.

[0089] In this embodiment, the initial hidden state can be retrieved and loaded using Formula 3 above, the input text, and the stored hidden states of each target text. In this way, during the pre-training phase, the server calculates and stores optimized initial hidden states for the frequently input target text. During inference, the server directly retrieves and loads the initial hidden states, reducing computational overhead.

[0090] like Figure 2 FIG. 1 is a schematic diagram of a process for determining an initial hidden state provided in an embodiment of the present application, which may include the following steps:

[0091] S201: Determine whether there is a historical context related to the input text.

[0092] If yes, execute the following S202; if no, execute the following S205.

[0093] S202: Calculate the text similarity between the historical context and the input text.

[0094] S203: Determine whether the text similarity is greater than a set similarity.

[0095] If yes, execute the following S204; if no, execute the following S205.

[0096] S204: Encode the historical context to obtain an encoding result, and use the encoding result as the initial hidden state.

[0097] S205: Determine an initial hidden state based on the stored hidden states corresponding to each target text and the input text.

[0098] Regarding the specific implementation process of the above S201 to S205, reference can be made to the above embodiments, which will not be repeated here.

[0099] S103: Using the neural network model, according to the inference pattern, the key-value pair reuse result and the inference parameter, determine the candidate output representation of the input text in the initial hidden state.

[0100] Here, the candidate output representation is used to indicate the output representation (i.e., output Token) that may be used to answer the input text after reasoning about the input text.

[0101] In specific implementation, a neural network model can be used to use the determined reasoning mode to reason about the input text in the initial hidden state, and the historical key-value pairs indicated by the key-value pair reuse results and the various parameters indicated by the reasoning parameters can be used during the reasoning process to obtain a candidate output Token for the input text.

[0102] S104: Generate a target output representation that matches the input text according to the sampling temperature that matches the importance of the candidate output representation, and return to the step of determining the initial hidden state of the neural network model and the key-value pair reuse result until each target output representation that matches the input text is generated to obtain the answer text corresponding to the input text.

[0103] Here, the target output representation is the output representation (output token) that actually answers the input text. Sampling temperature controls the diversity of the generated output. This application uses an adaptive sampling strategy to adjust the sampling temperature of each candidate output token, improving the quality of the target output representation while accelerating inference. The answer text is the text used to answer the input text.

[0104] In specific implementation, since the neural network model generates target output tokens one by one step at a time, that is, it generates the first target output token first, then generates the second, third, and so on, until the last target output token is obtained, so when generating each target output token, the server can determine the importance of the candidate output representation currently generated, and use the preset importance mapping function to determine the sampling temperature that matches the candidate output representation currently generated according to the importance and the preset sampling temperature, thereby adjusting the decoding parameters. Then, using the sampling temperature, candidate output tokens, input text and other information, a target output token for the input text is generated. In this way, by setting the sampling temperature based on the importance of the token, the sampling temperature can be lowered for key token positions (such as proper nouns) to improve the generation quality, and the sampling temperature can be increased for redundant token positions (such as stop words) to speed up the inference speed.

[0105] For example, the sampling temperature may be determined according to the following formula 1:

[0106]

[0107] in, represents the i-th candidate output representation x i importance, f represents the preset importance mapping function, T represents the preset sampling temperature, T′ represents x i The sampling temperature.

[0108] After obtaining the current target output representation, if it is the last target output representation, the answer text corresponding to the input text can be determined based on the text corresponding to each generated target output representation. If the current target output representation is not the last target output representation, the process returns to step S102 to generate the next target output representation until each target output representation matching the input text is generated. The answer text corresponding to the input text is determined based on the text corresponding to each target output representation.

[0109] In this way, based on the text length and semantic complexity, the most suitable inference mode and inference parameters can be determined for the input text, thereby dynamically optimizing the optimal computational process. This avoids wasting inference resources by using a fixed computational process and all parameters for all input texts using the neural network model. By determining an initial hidden state suitable for the input text and starting inference from this initial hidden state, the problem of low inference efficiency caused by the model inferring from a zero state can be avoided. By reusing key-value pair cache fragments, redundant computations during the inference process can be effectively reduced, thereby improving inference efficiency. By determining the sampling temperature based on the importance of the candidate output and using the sampling temperature to generate the target output representation, different sampling temperatures can be set for different candidate output representations, thereby flexibly controlling the diversity of target output representation generation. In summary, by selecting different inference modes, inference parameters, initial hidden states, key-value pair reuse results, sampling temperatures, and other information used in model inference for different input texts and controlling the neural network model to perform inference based on this information, the present application can effectively prioritize inference paths, improve inference speed, and reduce resource waste.

[0110] In one embodiment, the above S103 can be implemented according to the following steps:

[0111] S103-1: Using the neural network model, determine the initial attention weight of each input representation of the input text in the initial hidden state according to the inference mode, key-value pair reuse results and decoding step size.

[0112] Here, the input representation can be each input token of the input text, and the initial attention weight is the attention weight determined by the neural network model for each input token.

[0113] In specific implementation, a neural network model can be used to determine the initial attention weights corresponding to each input representation in the initial hidden state using the historical key-value pairs indicated by the key-value pair reuse results, the decoding step size in the inference parameters, and the attention calculation amount according to the determined inference mode.

[0114] S103-2: Sort the initial attention weights in descending order to obtain a sorting order.

[0115] In specific implementation, for each input token, the initial attention weights of the input token can be sorted in descending order to obtain the sorting order.

[0116] Optionally, the initial attention weights of all input tokens can be sorted in descending order to obtain the sorting order.

[0117] S103-3: The initial attention weight whose sorting order is less than or equal to the weight screening parameter is used as the target attention weight represented by the input.

[0118] Taking the weight screening parameter K in the inference parameter as an example, the initial attention weights whose sorting order is less than or equal to K can be screened out and used as the target attention weight of the input token. This can achieve dynamic attention weight pruning, ignore low-weight parts, and reduce the computational burden during the inference process.

[0119] In this way, the attention weights can be pruned through the dynamic attention mechanism. Combined with the above-mentioned adaptive sampling strategy to adjust the sampling temperature, the intermediate calculation path in the neural network model inference process can be optimized, thereby reducing invalid calculations and improving inference efficiency.

[0120] S103-4: Determine candidate output representations based on the target attention weight and input representation.

[0121] For example, the neural network model can process the screened target attention weights and the feature vector of each input representation through multiple layers of Transformer results to obtain the final hidden state, and use the final hidden state to determine the candidate output representation.

[0122] In one embodiment, the above S103 can be implemented according to the following steps A to C:

[0123] Step A: Determine the inference time of the neural network model under different inference paths.

[0124] Here, different reasoning paths may be the combination relationship between the sampling temperature, the attention weight, and the historical key-value pairs indicated by the key-value pair reuse result.

[0125] During specific implementation, in order to further improve the reasoning speed, the present application can also adopt a dynamic reasoning strategy based on search optimization to search for the optimal decoding path. For example, the server can use a random search method to randomly search the sampling temperature, attention weight and historical key-value pairs, thereby randomly initializing different reasoning paths and calculating the reasoning time t of each reasoning path. The reasoning path with the shortest reasoning time is selected as the model reasoning path. In this way, the optimal decoding parameter set can be determined through offline calculation, and the decoding parameter set indicates various combinations of decoding parameters, such as the combination of attention weight, sampling temperature, and historical KV cache.

[0126] Step B: Determine the computing resources allocated to each input representation based on the importance of each input representation of the input text and the importance of the generated target output representation.

[0127] Here, the present application also adopts an intelligent resource allocation strategy to allocate different computing resources to different input tokens.

[0128] Specifically, for each input token in the input text, the importance of the input token and the generated target output token can be determined based on their attention weights and semantic complexity. Different computing resources are then allocated to the input tokens based on their importance. Prior to outputting the first target output token, the number of generated target output tokens is 0; prior to outputting the second target output token, the number of generated target output tokens is 1, and so on. As the reasoning process progresses, the number of generated target output tokens gradually increases. For example, more important input tokens can be assigned higher computing resources to ensure computational accuracy, while less important input tokens can be allocated fewer computing resources, such as through quantization, to reduce the computational burden.

[0129] For example, the importance of the input token and the generated target output token can be determined according to the following formula 2:

[0130]

[0131] in, Indicates that the i-th input represents x i The importance of the generated target output representation, AttenScore(x i ) represents x i The target attention weight, λ is the preset parameter, PPL(x i ) represents x1~x iwhere x1 represents the first input representation or the first generated target output representation.

[0132] Step C: Utilize the neural network model to use the computing resources allocated for each input representation, follow the inference path with the least inference time, and determine the candidate output representation of the input text in the initial hidden state according to the inference mode, key-value pair reuse results and inference parameters.

[0133] In specific implementation, a neural network model can be used to determine the candidate output representation of the input text in the initial hidden state according to the inference parameters, using the computing resources allocated to each input representation in accordance with the determined inference mode and the combination of historical key-value pairs indicated by the inference path with the least inference time, target attention weights and sampling temperatures.

[0134] Optionally, when using a neural network model for reasoning, resources can also be scheduled according to the computing architecture of the server. Among them, the computing architecture is used to indicate the distribution of computing resources of the server. For example, when the server adopts a tensor processing unit (TPU) + graphics processing unit (GPU) architecture, computationally intensive tasks (such as Attentrion calculations) in the neural network model reasoning process can be assigned to high-performance GPUs for processing, and low-computational requirements tasks (such as multi-layer perceptron (MLP) computing tasks) can be assigned to low-power TPUs for processing. It is understandable that if the server only adopts a GPU architecture and does not deploy a TPU, all tasks need to be assigned to the GPU for processing. In this way, if the server adopts a heterogeneous computing architecture during the reasoning process, the reasoning efficiency can be further improved by combining the TPU / GPU resource scheduling strategy.

[0135] In one embodiment, the present application also provides a mechanism-based dynamic adjustment strategy for optimizing the determined inference parameters during the process of generating the target output representation, thereby improving inference efficiency. Specifically, before executing the step of returning the initial hidden state of the determined neural network model and the key-value pair reuse result, the following steps T1 to T3 are also included:

[0136] T1: Determines the time and attention computation proportion to generate the target output representation.

[0137] Here, the present application can monitor the output quality in real time during the inference process of the neural network model and dynamically adjust the inference path. The dynamic adjustment steps can include perplexity detection and adaptive bar hum and iterative optimization of the inference path.

[0138] In a specific implementation, before generating each target output representation and before generating the next target output representation, the time when the most recently generated target output representation and the proportion of attention calculation when generating the most recently generated target output representation can be determined.

[0139] T2: Adjust the key-value reuse rate based on time. The key-value reuse rate is used to adjust the proportion of reused key-value pairs in the key-value reuse result used in the next calculation of the candidate output representation.

[0140] In specific implementations, the key-value pair reuse rate can be adjusted based on the time taken to generate the target output representation and a time threshold. For example, if the time taken to generate the target output representation exceeds the time threshold, indicating that the decoding phase is taking a long time, the key-value pair reuse rate can be increased to reduce the time for the next decoding. The key-value pair reuse rate is used to adjust the proportion of reused key-value pairs in the key-value pair reuse results used in the next calculation of the candidate output representation. Since the generation of the target output representation depends on the candidate output representation, after the proportion of reused key-value pairs used in the next calculation of the candidate output representation is adjusted, the proportion of key-value pairs used in the calculation of the next target output representation is also adjusted accordingly. In other words, if the generation time of the current target output representation is long, the key-value pair reuse rate can be increased to increase the proportion of historical key-value pairs used in the generation of the next target output representation. If the time taken to generate the target output representation is not greater than the time threshold, indicating that the decoding phase is taking an ideal time, the key-value pair reuse rate used in the previous target output representation generation can be continued.

[0141] T3: Adjust the inference parameters based on the semantic complexity of the generated target output representation, the semantic complexity of the input text, and the attention calculation ratio.

[0142] During specific implementation, the output text composed of the various target output representations that have been generated can be determined, and the perplexity calculation module can be used to perform perplexity calculation on the output text to obtain the semantic complexity of the output text. Then, when the semantic complexity of the output text is greater than the semantic complexity of the input text, the weight screening parameter in the reasoning parameter can be increased to stabilize the quality of the target output representation generated next time. When the semantic complexity of the output text is not greater than the semantic complexity of the input text, the current weight screening parameter can continue to be used to generate the next target output representation. At the same time, if the attention calculation ratio is higher than the set ratio, the calculation accuracy in the reasoning parameter can be appropriately reduced. If the attention calculation ratio is not higher than the set ratio, the current calculation accuracy can be used to calculate the generation of the next target output representation.

[0143] This application provides an efficient neural network model inference optimization method that can dynamically adjust the inference path based on input features, historical context, and computational feedback, reducing computational costs and improving inference quality and speed. This application is suitable for large-model inference tasks such as natural language processing and computer vision. It can improve inference speed, optimize computing resource allocation, and increase the throughput and service quality of the inference system without requiring large-scale retraining.

[0144] like Figure 3 The flowchart shown is a specific implementation process of a text processing method provided in an embodiment of the present application, which may include the following steps:

[0145] S301: Input data feature analysis and classification processing.

[0146] Specifically, the input data is input text, and feature analysis can be performed on the input text to extract multiple key indicators, including text length, semantic complexity, and matching degree of key-value pair cache fragments.

[0147] Exemplarily, it may specifically include text length calculation and classification processing:

[0148] Assume that the input text X includes N input tokens and the text length L(X)=N; if L(X)<L Threshold , we can determine that the input text is a short text, and then we can use some network layer reasoning modes as the reasoning modes we need to use, such as reducing the number of Transformer layers or using quantitative calculations to improve reasoning and response speed; if L(X)≥L Threshold , it can be determined that the input text is a long text, and the full network layer reasoning mode can be used as the reasoning mode to ensure the reasoning accuracy.

[0149] Semantic complexity calculation and optimization strategy:

[0150] Call the perplexity calculation module to measure the semantic complexity of the input text. If PPL(X)<PPL Threshold , then the text structure is considered to be relatively simple and the input text belongs to simple text input, so the allocation of computing resources can be reduced, such as reducing the weight screening parameters, decoding step length and attention calculation amount that match the simple text input. If PPL(X)≥PPL Threshold , it is considered that the text structure is relatively complex and the input text belongs to complex text input. Therefore, a stronger decoding strategy can be adopted for the input text, increasing computing resources, improving the weight screening parameters that match the complex text input (that is, improving the attention weight sampling quality of TPO-K), decoding step size, and attention calculation amount.

[0151] Historical KV Cache matching degree calculation:

[0152] Assume that the historical fragment in KV Cache is represented as H = {h1, h2, ... h m}, the feature vector of the input text is represented as V(X), and the cosine similarity Sim(X, H) is calculated: if Sim(V(X), H)> the preset matching degree, the historical KV Cache is reused to reduce the amount of calculation; if Sim(X, H)≤ the preset matching degree, the historical KV Cache is not reused.

[0153] S302: Search for the optimal initial hidden state.

[0154] In order to improve the inference convergence speed, in the model decoding stage, the optimized initial hidden state is used to avoid initialization from zero. The specific method is as follows:

[0155] Initialization based on historical context: Let the historical context of the current input text C = {X1, X2, ... X m}, use the Transformer encoder to encode the historical context of the input text and obtain the encoding result h c = TransformerEncoder(C). Let the initial hidden state = h c , thereby reducing the early convergence time of the model.

[0156] Pre-computation initialization: During the pre-training phase, the hidden states that meet the set requirements (such as the best computational performance) are stored for the target text under common input modes, and the initial hidden states are directly retrieved and loaded during inference.

[0157] If there is no available historical context and the initial hidden state is not found, the initial hidden state can be set to all zeros.

[0158] S303: Optimize the intermediate path.

[0159] Dynamically adjust the calculation path during the neural network model decoding phase to reduce invalid calculations and improve inference efficiency, including:

[0160] Adaptive sampling strategy: Set the sampling temperature T to control the diversity of generation and dynamically adjust T of the candidate output representation: lower T at key token positions to improve generation quality, and increase T at redundant token positions to speed up inference.

[0161] Dynamic Attention Pruning: Calculate the initial Attention weight of each input representation of the input text, retain only the Top-K largest attention weight calculations, and ignore the low-weight parts to reduce the computational burden.

[0162] S304: Dynamic reasoning strategy based on search optimization.

[0163] Specifically, a search algorithm can be used to optimize the decoding path and improve inference efficiency, including:

[0164] Random search: Use a search optimization algorithm to randomly initialize different inference paths, calculate the inference time t, and select the inference path with the shortest inference time mint as the model inference path.

[0165] S305: Intelligent computing resource allocation.

[0166] The server dynamically allocates computing resources based on the importance of tokens. Specifically, it can dynamically allocate computing resources based on the importance of tokens to improve throughput. More important input tokens are given more computing resources to ensure computational accuracy, while less important input tokens are allocated fewer computing resources, such as through quantization, to reduce the computational burden.

[0167] Multi-GPU scheduling: During the inference process, if the server adopts a heterogeneous computing architecture, it combines the TPU / GPU resource scheduling strategy. For example, compute-intensive tasks (such as Attention calculation) are assigned to high-performance GPU processing; low-computational-demand tasks (such as MLP calculation) are scheduled to low-power TPU processing.

[0168] S306: Dynamic adjustment based on feedback mechanism.

[0169] The server monitors the output quality in real time during the inference process and dynamically adjusts the inference path.

[0170] Specifically, perplexity monitoring and adaptive adjustment can be performed.

[0171] If the perplexity of the output text composed of the currently generated target output representations increases significantly, the server adjusts the Top-K sampling strategy to stabilize the generation quality. Furthermore, the server iteratively optimizes the inference path. The server continuously records the computation time during the inference process and optimizes the computation path based on historical data. For example, if the Attention calculation accounts for a high proportion, the computation accuracy is appropriately reduced. If the decoding phase takes a long time, the KV Cache reuse rate is increased.

[0172] In this way, this application provides a large-model inference acceleration method based on dynamic optimization and intelligent scheduling. This method focuses on core technologies such as input feature analysis, calculation path optimization, intelligent resource allocation and feedback mechanism adjustment to optimize inference efficiency, reduce computing overhead, and improve throughput.

[0173] Corresponding to the aforementioned embodiments of the text processing method, the present application also provides embodiments of a text processing device.

[0174] The embodiment of the text processing device of the present application can be applied to a computer device. The embodiment of the device can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the computer device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running it. From the hardware level, if Figure 4 As shown, this is a hardware structure diagram of the computer device where the text processing device 500 of this application is located. Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, the computer device where the apparatus in the embodiment is located may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0175] Please refer to Figure 5 , is a structural diagram of a text processing device provided in an embodiment of the present application, comprising:

[0176] A first determination module 501 is used to determine the reasoning mode and reasoning parameters of the neural network model according to the text length and semantic complexity of the input text;

[0177] A second determination module 502 is configured to determine an initial hidden state of the neural network model and a key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context;

[0178] A third determination module 503 is configured to determine, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter;

[0179] A generation module 504 is used to generate a target output representation that matches the input text according to a sampling temperature that matches the importance of the candidate output representation, and return to the step of determining the initial hidden state of the neural network model and the key-value pair multiplexing result until each target output representation that matches the input text is generated to obtain an answer text corresponding to the input text.

[0180] In a possible implementation, the first determining module 501, when determining the reasoning mode and reasoning parameters of the neural network model according to the text length and semantic complexity of the input text, is configured to:

[0181] Selecting the inference mode from preset modes according to the text length and the length threshold of the input text; the preset modes include full network layer inference and partial network layer inference;

[0182] According to the semantic complexity, a weight screening parameter and a decoding step size for the input representation are determined, and the weight screening parameter and the decoding step size are used as the inference parameters.

[0183] In one possible implementation, the third determination module 503, when determining the candidate output representation of the input text in the initial hidden state using the neural network model according to the inference mode, the key-value pair reuse result, and the inference parameter, is configured to:

[0184] Determining, using the neural network model, an initial attention weight for each input representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the decoding step size;

[0185] Sort the initial attention weights in descending order to obtain a sorting order;

[0186] The initial attention weight whose sorting order is less than or equal to the weight screening parameter is used as the target attention weight of the input representation;

[0187] Determine the candidate output representation based on the target attention weight and the input representation.

[0188] In one possible implementation, the third determination module 503, when determining the candidate output representation of the input text in the initial hidden state using the neural network model according to the inference mode, the key-value pair reuse result, and the inference parameter, is configured to:

[0189] Determining the inference time of the neural network model under different inference paths;

[0190] Determining computing resources to allocate to each input representation based on the importance of each input representation of the input text and the importance of the generated target output representation;

[0191] The neural network model is used to use the computing resources allocated to each input representation, and according to the reasoning path with the shortest reasoning time, the candidate output representation of the input text is determined in the initial hidden state according to the reasoning mode, the key-value pair reuse result and the reasoning parameters.

[0192] In a possible implementation, the apparatus further includes an adjustment module 505, which, before returning the step of determining the initial hidden state of the neural network model and the key-value pair reuse result, is configured to:

[0193] Determining the time and attention computational proportion for generating the target output representation;

[0194] Adjusting a key-value pair reuse rate according to the time; the key-value pair reuse rate is used to adjust the proportion of reused key-value pairs in a key-value pair reuse result used in the next calculation of the candidate output representation;

[0195] The inference parameters are adjusted according to the semantic complexity corresponding to the generated target output representation, the semantic complexity of the input text, and the attention calculation ratio.

[0196] In a possible implementation, the second determining module 502, when determining the initial hidden state of the neural network model and the key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context, is configured to:

[0197] When the text similarity between the historical context and the input text is greater than a set similarity, determining a matching degree between the input text and the key-value pair cache segment based on the key-value pair cache segment of the historical context and a feature vector of the input text;

[0198] The key-value pair reuse result is determined according to the matching degree, and the initial hidden state is determined according to the encoding result of the historical context.

[0199] In a possible implementation, the second determining module 502 is further configured to:

[0200] If the historical context does not exist or the text similarity is not greater than the set similarity, the key-value pair reuse result is determined to be empty, and the initial hidden state is determined based on the hidden states corresponding to the stored target texts and the input text; wherein the target text includes text whose input frequency is higher than the set frequency during the model training process.

[0201] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0202] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0203] Embodiments of the subject matter and functional operations described in this specification may be implemented in the following: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or a combination of one or more of them. Embodiments of the subject matter described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to be executed by a data processing device or to control the operation of the data processing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagation signal, such as a machine-generated electrical, optical, or electromagnetic signal, which is generated to encode information and transmit it to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

[0204] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform the corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0205] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory and / or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, or the computer will be operably coupled to such mass storage devices to receive data from them or to transmit data to them, or both. However, a computer does not necessarily have such devices. In addition, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0206] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0207] Although this specification includes many specific implementation details, these should not be interpreted as limiting the scope of any invention or the scope of protection claimed, but are mainly used to describe the features of specific embodiments of specific inventions. Certain features described in multiple embodiments within this specification may also be implemented in combination in a single embodiment. On the other hand, the various features described in a single embodiment may also be implemented separately in multiple embodiments or in any suitable sub-combination. In addition, although features may work in certain combinations as described above and even initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may point to a sub-combination or a variation of the sub-combination.

[0208] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that these operations be performed in the particular order shown or performed sequentially, or that all illustrated operations be performed to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product, or packaged into multiple software products.

[0209] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the particular order shown or sequential sequence to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0210] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: The method comprises: Determine the reasoning mode and reasoning parameters of the neural network model based on the text length and semantic complexity of the input text; Determining an initial hidden state of the neural network model and a key-value pair reuse result based on the historical context of the input text and the key-value pair cache fragment of the historical context; Determining, using the neural network model, candidate output representations of the input text in the initial hidden state according to the inference pattern, the key-value pair reuse result, and the inference parameter; According to the sampling temperature that matches the importance of the candidate output representation, a target output representation that matches the input text is generated, and the step of determining the initial hidden state of the neural network model and the key-value pair multiplexing result is returned until each target output representation that matches the input text is generated, and an answer text corresponding to the input text is obtained.

2. The method according to claim 1, characterized in that Determining the reasoning mode and reasoning parameters of the neural network model based on the text length and semantic complexity of the input text includes: Selecting the inference mode from preset modes according to the text length and the length threshold of the input text; the preset modes include full network layer inference and partial network layer inference; According to the semantic complexity, a weight screening parameter and a decoding step size for the input representation are determined, and the weight screening parameter and the decoding step size are used as the inference parameters.

3. The method according to claim 2, characterized in that The determining, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter, includes: Determining, using the neural network model, an initial attention weight for each input representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the decoding step size; Sort the initial attention weights in descending order to obtain a sorting order; The initial attention weight whose sorting order is less than or equal to the weight screening parameter is used as the target attention weight of the input representation; Determine the candidate output representation based on the target attention weight and the input representation.

4. The method according to claim 1, wherein The determining, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter, includes: Determining the inference time of the neural network model under different inference paths; Determining computing resources to allocate to each input representation based on the importance of each input representation of the input text and the importance of the generated target output representation; The neural network model is used to use the computing resources allocated to each input representation, and according to the reasoning path with the shortest reasoning time, the candidate output representation of the input text is determined in the initial hidden state according to the reasoning mode, the key-value pair reuse result and the reasoning parameters.

5. The method according to claim 3, characterized in that Before returning the step of determining the initial hidden state of the neural network model and the key-value pair reuse result, the method further includes: Determining the time and attention computational proportion for generating the target output representation; Adjusting a key-value pair reuse rate according to the time; the key-value pair reuse rate is used to adjust the proportion of reused key-value pairs in a key-value pair reuse result used in the next calculation of the candidate output representation; The inference parameters are adjusted according to the semantic complexity corresponding to the generated target output representation, the semantic complexity of the input text, and the attention calculation ratio.

6. The method according to claim 1, wherein The determining, based on the historical context of the input text and the key-value pair cache fragments of the historical context, the initial hidden state of the neural network model and the key-value pair reuse result includes: When the text similarity between the historical context and the input text is greater than a set similarity, determining a matching degree between the input text and the key-value pair cache segment based on the key-value pair cache segment of the historical context and a feature vector of the input text; The key-value pair reuse result is determined according to the matching degree, and the initial hidden state is determined according to the encoding result of the historical context.

7. The method according to claim 6, characterized in that The method further comprises: If the historical context does not exist or the text similarity is not greater than the set similarity, the key-value pair reuse result is determined to be empty, and the initial hidden state is determined based on the hidden states corresponding to the stored target texts and the input text; wherein the target text includes text whose input frequency is higher than the set frequency during the model training process.

8. A text processing device, characterized in that: The device comprises: A first determination module is used to determine the reasoning mode and reasoning parameters of the neural network model according to the text length and semantic complexity of the input text; A second determination module is configured to determine an initial hidden state of the neural network model and a key-value pair reuse result based on a historical context of the input text and a key-value pair cache segment of the historical context; a third determining module, configured to determine, using the neural network model, a candidate output representation of the input text in the initial hidden state according to the inference mode, the key-value pair reuse result, and the inference parameter; A generation module is used to generate a target output representation that matches the input text according to a sampling temperature that matches the importance of the candidate output representation, and return to the step of determining the initial hidden state of the neural network model and the key-value pair multiplexing result until each target output representation that matches the input text is generated to obtain an answer text corresponding to the input text.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images