Large model reasoning method for user request and server

Abstract

The invention discloses a large model reasoning method for a user request and a server, and relates to the technical field of large model application, and the method comprises the steps: inputting an obtained request text and a target statement into a preset language model for reasoning processing, if it is judged that the request text can be matched with the historical prefix key-value cache in the host memory, determining the historical prefix key-value cache as a preliminary pre-filling reasoning result; performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target statement to obtain a pre-filling reasoning result; performing recursive reasoning processing according to the pre-filling reasoning result to obtain a recursive reasoning result; the recursive reasoning result is output to the user side, the historical prefix key-value cache is stored through the host memory, and the corresponding historical prefix key-value cache does not need to be reasoned repeatedly, so that the reasoning efficiency of the request text is improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of large model application technology, and in particular to a large model reasoning method and server for user requests. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and deep learning, users have an increasingly urgent need for large artificial intelligence models. Large models refer to machine learning models with large-scale parameters and complex computing structures. In particular, with the explosion of AI intelligent question-answering, the attention paid to large language models has soared.

[0003] In the related art, the request text proposed by the user is inferred through the GPU memory to obtain an inference result containing prefix data. However, in the related art, after obtaining the inference result containing prefix data, the GPU memory will clear the prefix data, so that the prefix data needs to be repeatedly inferred in subsequent inferences, resulting in a long inference time and reducing the user experience. Summary of the invention

[0004] The present application provides a large model inference method and server for user requests, so as to at least solve the problem in the related art that after obtaining the inference result containing prefix data, the GPU video memory will clear the prefix data, so that the prefix data needs to be repeatedly inferred in subsequent inference, resulting in excessively long inference time and reduced user experience.

[0005] The present application provides a large model reasoning method for user requests, including:

[0006] Receive the request text sent by the user;

[0007] Performing vector conversion processing on the request text to obtain a corresponding request vector;

[0008] Determining a corresponding target sentence from a pre-created knowledge base according to the request vector;

[0009] The request text and the target sentence are input into a preset language model for inference processing to perform the following steps:

[0010] Determine whether the request text can match a pre-stored historical prefix key-value cache in the host memory, wherein the historical prefix key-value cache is pre-filled and inferred based on the historical request text;

[0011] If the request text can match the historical prefix key-value cache in the host memory, the historical prefix key-value cache is determined as a preliminary pre-filled inference result;

[0012] Performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a pre-filling reasoning result;

[0013] Performing recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result;

[0014] The recursive reasoning result is output to the user end.

[0015] The present application also provides a server, including: a memory for storing a computer program; a processor for implementing any of the steps of the above-mentioned large model inference method for user requests when executing the computer program.

[0016] The present application also provides a computer-readable storage medium, in which a computer program is stored, wherein when the computer program is executed by a processor, the steps of any of the above-mentioned large model inference methods for user requests are implemented.

[0017] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned large model reasoning methods for user requests.

[0018] The large model reasoning method and server for user requests provided in the embodiments of the present application perform the following steps by inputting the acquired request text and target sentence into a preset language model for reasoning processing: after determining that the request text can match the historical prefix key-value cache in the host memory, determining the historical prefix key-value cache as a preliminary pre-filled reasoning result; performing pre-filled reasoning processing according to the request text, the preliminary pre-filled reasoning result and the target sentence to obtain a pre-filled reasoning result; performing recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result; outputting the recursive reasoning result to the user end, and storing the historical prefix key-value cache through the host memory, without repeatedly reasoning the corresponding historical prefix key-value cache, thereby improving the reasoning efficiency of the request text and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of a scenario of a large model reasoning method for user requests provided in an embodiment of the present application;

[0021] Figure 2Schematic diagram of the process of the large model inference method for user request provided in the embodiment of the present application Figure 1 ;

[0022] Figure 3 Schematic diagram of the process of the large model inference method for user request provided in the embodiment of the present application Figure 2 ;

[0023] Figure 4 A schematic diagram of the structure of a large model inference device for user request provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] It should be noted that, in the description of this application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0027] In recent years, with the rapid development of artificial intelligence and deep learning, users have an increasingly urgent demand for large artificial intelligence models. Large models refer to machine learning models with large-scale parameters and complex computing structures. These models are usually built with deep neural networks and have billions or even hundreds of billions of parameters. They are designed to improve the expressiveness and predictive performance of the model to handle more complex tasks and data. In particular, with the explosion of AI intelligent question-answering, the attention paid to large language models has soared. In the related art, the request text submitted by the user is inferred through the GPU video memory to obtain an inference result containing prefix data. However, in the related art, after obtaining the inference result containing prefix data, the GPU video memory will clear the prefix data, so that the prefix data needs to be repeatedly inferred in subsequent inferences, resulting in too long inference time, which reduces the user experience.

[0028] In order to solve the above technical problems, the embodiments of the present application propose the following technical concept: the inventor considers the obtained request text and the corresponding target sentence, and after determining that the request text can match the historical prefix key-value cache in the host memory, pre-fill reasoning processing and recursive reasoning processing are performed on the historical prefix key-value cache, the request text and the corresponding target sentence based on a preset language model to obtain pre-fill reasoning results and recursive reasoning results, and output the pre-filled reasoning results and recursive reasoning results, thereby improving the reasoning efficiency of the request text and improving the user experience.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0030] In conjunction with the specific application environment architecture or specific hardware architecture on which the execution of the large model reasoning method for user requests depends, the specific application environment architecture or specific hardware architecture is described herein. Figure 1 , Figure 1 A schematic diagram of a scenario of a large model inference method for user requests provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the scenario includes: a user terminal 101 and a server 102.

[0032] The user terminal 101 may be a mobile phone terminal or a personal computer terminal.

[0033] The server 102 may be an independent server or a cluster consisting of multiple servers.

[0034] The user terminal 101 sends the request text to the server 102 via the wireless network. The server 102 performs vector conversion processing on the request text to obtain a corresponding request vector; determines the corresponding target sentence from the pre-created knowledge base according to the request vector; inputs the request text and the target sentence into a preset language model for reasoning processing to perform the following steps: if it is determined that the request text can match the historical prefix key-value cache in the host memory, the historical prefix key-value cache is determined as a preliminary pre-filled reasoning result; performs pre-filled reasoning processing according to the request text, the preliminary pre-filled reasoning result and the target sentence to obtain a pre-filled reasoning result; performs recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result; and outputs the recursive reasoning result to the user terminal 101.

[0035] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0036] Figure 2 Schematic diagram of the process of the large model inference method for user request provided in the embodiment of the present application Figure 1 ,like Figure 2 As shown, the embodiment of the present application provides a large model reasoning method for user requests, and the method is described in detail as follows:

[0037] S201: Receive a request text sent by a user terminal.

[0038] S202: Perform vector conversion processing on the request text to obtain a corresponding request vector.

[0039] Specifically, the request text is vector-converted according to the encoder to obtain a corresponding request vector.

[0040] In this embodiment, the request vector is a method for describing the characteristics of an object by converting the request text into structured vectors so that a large model can understand and process the data, and these vectors represent the characteristics of the request text.

[0041] S203: Determine a corresponding target sentence from a pre-created knowledge base according to the request vector.

[0042] Specifically, the process of creating the knowledge base in step S203 specifically includes steps a to b:

[0043] Step a: receiving one or more documents sent by the user.

[0044] In this embodiment, the document may be company rules and regulations, notices and announcements, or other documents.

[0045] Step b: Create a knowledge base based on one or more documents.

[0046] Specifically, step S203 includes:

[0047] S2031: Determine the target document from the knowledge base according to the request vector.

[0048] S2032: Determine a corresponding target sentence from the target document according to the request vector.

[0049] In this embodiment, the target sentence may be a document fragment, a paragraph, or other combined sentences in the target document.

[0050] S204: Input the request text and the target sentence into a preset language model for inference processing to perform the following steps:

[0051] S205: Determine whether the request text can match a pre-stored historical prefix key-value cache in the host memory, wherein the historical prefix key-value cache is pre-filled and inferred based on the historical request text.

[0052] Specifically, a matching algorithm corresponding to the large model reasoning program is used to determine whether the request text can be matched to a pre-stored historical prefix key-value cache in the host memory.

[0053] Among them, the L2 cache provides temporary space for the inference program, and the L2 cache belongs to the GPU video memory.

[0054] Among them, L2 cache is the second-level cache of the CPU, located between the CPU and the main memory, and usually composed of static random access memory. The capacity of L2 cache is smaller than that of the main memory, but the exchange speed is faster. Its main function is to serve as a temporary memory between the CPU and the main memory, storing data that the CPU may access in the short term, so as to reduce the number of accesses to the main memory, thereby improving the data access speed.

[0055] Among them, CPU is the central processing unit.

[0056] Specifically, the storage process of the historical prefix key-value cache in step S205 specifically includes steps c to d:

[0057] Step c: Obtain multiple historical prefix key-value caches.

[0058] In this embodiment, the historical prefix key-value cache is a prefix kv cache.

[0059] Among them, kv cache is a technology used to optimize the inference performance of the Transformer model, especially in the autoregressive generation model. Its core function is to cache the Key and Value matrices to avoid repeated calculation of these matrices every time a new token is generated, thereby significantly improving the inference efficiency.

[0060] Step d: Storing multiple historical prefix key-value caches in host memory.

[0061] Specifically, step d specifically includes:

[0062] Step d1: each historical prefix matrix data is labeled according to a preset data format to obtain each labeled historical prefix key-value cache.

[0063] In this embodiment, the preset data format may be length, type, keyword, or other data formats.

[0064] Step d2: Generate tree structure data according to the historical prefix key-value cache after each annotation.

[0065] In this embodiment, each annotated historical prefix key-value cache corresponds to each tree node data in the tree structure data.

[0066] Step d3: Store the tree structure data into the host memory.

[0067] In this embodiment, the host memory is the server-side physical memory.

[0068] Specifically, step S205 includes:

[0069] S2051: Obtain tree structure data in the host memory, wherein the tree structure data includes one or more tree node data, and each tree node data corresponds to a pre-stored historical prefix key-value cache.

[0070] S2052: Obtain identification information corresponding to each historical prefix key-value cache.

[0071] In this embodiment, the identification information includes length, type, keyword and other identification information.

[0072] S2053: Determine whether the request text can match the corresponding identification information.

[0073] Specifically, key information of the request text is extracted, and it is determined whether corresponding identification information can be matched based on the key information.

[0074] Among them, the corresponding identification information is matched through GPU real-time reasoning.

[0075] Therefore, in combination with the above description, by storing the tree structure data including the historical prefix key-value cache in the host memory, and matching the identification information corresponding to the historical prefix key-value cache through GPU real-time reasoning, a corresponding hierarchical cache layout is constructed.

[0076] Among them, GPU video memory stands for graphics processing unit memory or video memory, which is a high-speed memory specially designed for GPU, mainly used to store data related to graphics rendering, such as textures, vertex data and frame buffers, etc. Video memory is usually used in graphics cards or integrated graphics solutions, and its design purpose is to meet the needs of large-scale parallel computing and provide fast data access channels.

[0077] Among them, GPU is a graphics processing unit.

[0078] S2054: If the request text can match the corresponding identification information, it is determined that the request text can match the historical prefix key-value cache in the host memory; if the request text cannot match the corresponding identification information, it is determined that the request text cannot match the historical prefix key-value cache in the host memory.

[0079] S206: If the request text can match the historical prefix key-value cache in the host memory, the historical prefix key-value cache is determined as a preliminary pre-filled inference result.

[0080] In addition, after step S206, steps e to j are also included:

[0081] Step e: Obtain the host memory usage threshold;

[0082] In this embodiment, the usage threshold may be any number among 10 times, 100 times, or 1000 times, or may be other numbers.

[0083] Step f: Count the historical prefix key-value cache to obtain the current count;

[0084] Step g: Obtain multiple historical counts in the host memory;

[0085] Step h: determine whether the total number of counts of multiple historical counts and the current count is greater than the usage threshold;

[0086] Step i: if it is determined that the total count quantity is greater than the usage threshold, then determining the target tree node to be filtered out in the tree structure data;

[0087] In this embodiment, the target tree node to be filtered out may be a tree node with the lowest number of uses, or may be another tree node.

[0088] Step j: Filter the target tree node from the host memory to update the host memory.

[0089] S207: Perform pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a pre-filling reasoning result.

[0090] Specifically, step S207 includes:

[0091] S2071: Perform pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a prefix key-value cache and an initial word unit.

[0092] In this embodiment, the prefill reasoning is the Prefill stage reasoning.

[0093] Prefill phase reasoning refers to a preprocessing phase before the model is formally reasoned during the large model reasoning process. The main purpose of this phase is to prefill or preheat the model to ensure that the model can respond to requests more quickly during the reasoning phase.

[0094] In this embodiment, the initial word is a token.

[0095] Among them, token is a word unit, which refers to the symbol used to represent a word or phrase in the process of natural language processing; a token can be a single word unit or a sequence of multiple word unit symbols.

[0096] In addition, after step S2071, steps k~l are also included:

[0097] Step k: Decode the initial word to obtain a corresponding character string.

[0098] Step 1: Output the string to the user end.

[0099] S2072: Determine the prefix key-value cache and the initial word as pre-filled inference results.

[0100] S208: Perform recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result.

[0101] Specifically, step S208 includes:

[0102] S2081: Continue to reason based on the initial word, recursively reasoning until a preset end marker is inferred.

[0103] In this embodiment, the recursive reasoning is the Decode stage reasoning.

[0104] Decode phase reasoning refers to the process in which the model continues to autoregressively generate subsequent tokens after generating the first token in the large model reasoning process until the stopping condition is met, such as generating a specific number of tokens or encountering a specific end tag.

[0105] In this embodiment, the preset end identifier is eos_token_id.

[0106] Specifically, the prefix key-value cache and the initial word unit are used as the input cache, and reasoning is continued according to the initial word unit, and recursive reasoning is performed until a preset end mark is inferred.

[0107] Recursive reasoning refers to the process of repeating reasoning by using the last inference word as the input for the next time.

[0108] S2082: Obtain multiple word units before the last word unit.

[0109] In addition, the plurality of word units and the last word unit are decoded to obtain corresponding character strings.

[0110] S2083: Determine each word as a recursive reasoning result.

[0111] S209: Output the recursive reasoning result to the user end.

[0112] Specifically, each character string corresponding to the recursive reasoning result is output to the user end.

[0113] In addition, each time a character string is generated, the character string can be output to the user end.

[0114] In summary, the large model reasoning method for user requests provided in this embodiment performs the following steps by inputting the obtained request text and target sentence into a preset language model for reasoning processing: after determining that the request text can match the historical prefix key-value cache in the host memory, the historical prefix key-value cache is determined as a preliminary pre-filled reasoning result; pre-filled reasoning processing is performed according to the request text, the preliminary pre-filled reasoning result and the target sentence to obtain a pre-filled reasoning result; recursive reasoning processing is performed according to the pre-filled reasoning result to obtain a recursive reasoning result; the recursive reasoning result is output to the user end, and the historical prefix key-value cache is stored in the host memory, without the need to repeatedly reason the corresponding historical prefix key-value cache, thereby improving the reasoning efficiency of the request text and improving the user experience.

[0115] In addition, the large model inference method for user requests provided in this embodiment forms a corresponding hierarchical cache layout by storing the tree structure data including the historical prefix key-value cache to the host memory, and matching the identification information corresponding to the historical prefix key-value cache through GPU real-time inference, so that the prefix key-value cache can be cached for a long time without affecting the reading and writing speed of the GPU, and the arrangement of each storage unit is more reasonable for the reasoning of subsequent requested texts.

[0116] Figure 3 Schematic diagram of the process of the large model inference method for user request provided in the embodiment of the present application Figure 2 In the embodiment of the present application, Figure 2 Based on the embodiment provided, a subsequent reasoning method for the case where the request text cannot be matched to the historical prefix key-value cache in the host memory after step S205 is described in detail. Figure 3 As shown, the method includes:

[0117] S301: Receive a request text sent by a user terminal.

[0118] S302: Perform vector conversion processing on the request text to obtain a corresponding request vector.

[0119] In this embodiment, the discussion on the request vector has been described in detail in step S202 and will not be repeated here.

[0120] S303: Determine a corresponding target sentence from a pre-created knowledge base according to the request vector.

[0121] In this embodiment, the discussion on the pre-created knowledge base and the target sentence has been described in detail in step S203 and will not be repeated here.

[0122] S304: Input the request text and the target sentence into a preset language model for inference processing to perform the following steps:

[0123] S305: Determine whether the request text can match a pre-stored historical prefix key-value cache in the host memory, wherein the historical prefix key-value cache is pre-filled and inferred based on the historical request text.

[0124] In this embodiment, the discussion about the host memory and the pre-stored historical prefix key-value cache has been described in detail in step S205, which will not be repeated here.

[0125] S306: If the request text cannot be matched to the historical prefix key-value cache in the host memory, the request text and the target sentence are input into the preset language model for pre-filling reasoning and recursive reasoning to obtain the corresponding reasoning result.

[0126] Specifically, step S306 includes:

[0127] S3061: Input the request text and the target sentence into a preset language model for pre-filling reasoning processing to obtain a prefix key-value cache and a first word.

[0128] In this embodiment, the discussion on pre-filled reasoning and word-grams has been described in detail in step S207 and will not be repeated here.

[0129] S3062: Determine the prefix key-value cache and the first word as pre-filled inference results.

[0130] S3063: Input the first word into a preset language model for recursive reasoning until a preset end mark is inferred.

[0131] In this embodiment, the discussion on recursive reasoning has been described in detail in step S208 and will not be repeated here.

[0132] S3064: Obtain multiple word units before the last word unit.

[0133] S3065: Determine each word as a recursive reasoning result.

[0134] S3066: Determine the pre-filled reasoning result and the recursive reasoning result as the reasoning result.

[0135] In addition, after step S306, the method further includes:

[0136] From the inference results, the prefix key-value cache corresponding to the request text is integrated into the tree structure data.

[0137] Specifically, from the pre-filled inference results of the inference results, the prefix key-value cache corresponding to the request text is integrated into the tree structure data.

[0138] S307: Output the inference processing result to the user end.

[0139] In summary, the large model inference method for user requests provided in this embodiment, if the request text cannot be matched to the historical prefix key-value cache in the host memory, the request text and the target sentence are input into the preset language model for pre-filled reasoning and recursive reasoning to obtain the corresponding reasoning result; the reasoning processing result is output to the user end, and the prefix key-value cache corresponding to the request text is integrated into the tree structure data, so that the tree structure data in the host memory can be supplemented in time for storage as the corresponding historical prefix key-value cache.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0141] Figure 4 A schematic diagram of the structure of a large model inference device for user request provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application also provides a large model inference device for user requests, including: a first receiving module 401, a first processing module 402, a first determination module 403, a second processing module 404, a first judgment module 405, a second determination module 406, a third processing module 407, a fourth processing module 408 and a first output module 409.

[0142] The first receiving module 401 is used to receive a request text sent by a user terminal;

[0143] A first processing module 402 is used to perform vector conversion processing on the request text to obtain a corresponding request vector;

[0144] A first determination module 403, configured to determine a corresponding target sentence from a pre-created knowledge base according to the request vector;

[0145] The second processing module 404 is used to input the request text and the target sentence into a preset language model for inference processing to perform the following steps:

[0146] A first judgment module 405 is used to judge whether the request text can match a pre-stored historical prefix key-value cache in the host memory, wherein the historical prefix key-value cache is pre-filled and inferred based on the historical request text;

[0147] A second determination module 406 is configured to determine the historical prefix key-value cache as a preliminary pre-filled inference result if the request text can be matched to the historical prefix key-value cache in the host memory;

[0148] A third processing module 407 is used to perform pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a pre-filling reasoning result;

[0149] A fourth processing module 408, configured to perform recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result;

[0150] The first output module 409 is used to output the recursive reasoning result to the user end.

[0151] In a possible implementation, the determination module 405 specifically includes:

[0152] A first acquisition unit is used to acquire tree structure data in the host memory, wherein the tree structure data includes one or more tree node data, each tree node data corresponds to a pre-stored historical prefix key-value cache;

[0153] A second acquisition unit is used to acquire identification information corresponding to each historical prefix key-value cache;

[0154] A judging unit, used to judge whether the request text can match the corresponding identification information;

[0155] A determination unit is used to determine whether the request text can match the historical prefix key-value cache in the host memory if the request text can match the corresponding identification information; if the request text cannot match the corresponding identification information, determine whether the request text cannot match the historical prefix key-value cache in the host memory.

[0156] In a possible implementation, the third processing module 407 specifically includes:

[0157] A processing unit, configured to perform pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a prefix key-value cache and an initial word unit;

[0158] A determining unit is used to determine the prefix key-value cache and the initial word as pre-filled inference results.

[0159] In a possible implementation manner, the device further includes:

[0160] A fifth processing module, configured to decode the initial word element to obtain a corresponding character string;

[0161] The second output module is used to output the character string to the user terminal.

[0162] In a possible implementation, the fourth processing module 408 specifically includes:

[0163] An inference unit, used to continue inference based on the initial word element, recursively inferring until a preset end mark is inferred;

[0164] An acquisition unit, used for acquiring multiple tokens before the last token;

[0165] A determination unit is used to determine each word-gram as the recursive reasoning result.

[0166] In a possible implementation manner, the device further includes:

[0167] A first acquisition module is used to acquire multiple historical prefix key-value caches;

[0168] The storage module is used to store the multiple historical prefix key-value caches in the host memory.

[0169] In a possible implementation, the storage module specifically includes:

[0170] A labeling unit, used for labeling each historical prefix matrix data according to a preset data format to obtain each labeled historical prefix key-value cache;

[0171] A generating unit, configured to generate tree structure data according to the annotated historical prefix key-value cache;

[0172] A storage unit is used to store the tree structure data in the host memory.

[0173] In a possible implementation manner, the device further includes:

[0174] a sixth processing module, configured to input the request text and the target sentence into the preset language model for pre-filling reasoning and recursive reasoning to obtain a corresponding reasoning result if the request text cannot be matched to the historical prefix key-value cache in the host memory;

[0175] The third output module is used to output the reasoning processing result to the user end.

[0176] In a possible implementation, the device further includes:

[0177] An integration module is used to integrate the prefix key-value cache corresponding to the request text into the tree structure data from the inference result.

[0178] In a possible implementation manner, the device further includes:

[0179] A second acquisition module is used to acquire a usage threshold of the host memory;

[0180] A counting module, used for counting the historical prefix key-value cache to obtain a current count;

[0181] A third acquisition module, used for acquiring a plurality of historical counts in the host memory;

[0182] A second judgment module is used to judge whether the total count quantity of the multiple historical counts and the current count is greater than the usage threshold;

[0183] a third determination module, for determining a target tree node to be filtered out in the tree structure data if it is determined that the total count quantity is greater than the usage threshold;

[0184] A filtering module is used to filter the target tree node from the host memory to update the host memory.

[0185] In a possible implementation manner, the device further includes:

[0186] A second receiving module, used to receive one or more documents sent by the user terminal;

[0187] A creation module is used to create a knowledge base according to the one or more documents.

[0188] In a possible implementation, the first determining module 403 specifically includes:

[0189] A first determining unit, configured to determine a target document from the knowledge base according to the request vector;

[0190] The second determining unit is configured to determine a corresponding target sentence from the target document according to the request vector.

[0191] For the description of the features in the embodiments corresponding to the large model reasoning device for user requests, please refer to the relevant description of the embodiments corresponding to the large model reasoning method for user requests, which will not be repeated here.

[0192] Figure 5 This is a schematic diagram of the server structure provided for this application. Figure 5 As shown, the server provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the server also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus.

[0193] In the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes the above-mentioned large model reasoning method embodiment for user requests.

[0194] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0195] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0196] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0197] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0198] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned large model reasoning method embodiments for user requests when running.

[0199] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0200] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above-mentioned large model reasoning method embodiments for user requests are implemented.

[0201] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, the non-volatile computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, implementing the steps in any of the above-mentioned large model reasoning method embodiments for user requests.

[0202] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0203] The above is a detailed introduction to a large model reasoning method for user requests provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A large model reasoning method for user requests, characterized in that: include: Receive the request text sent by the user; Performing vector conversion processing on the request text to obtain a corresponding request vector; Determining a corresponding target sentence from a pre-created knowledge base according to the request vector; The request text and the target sentence are input into a preset language model for inference processing to perform the following steps: Determine whether the request text can match a pre-stored historical prefix key-value cache in the host memory, wherein the historical prefix key-value cache is pre-filled and inferred based on the historical request text; If the request text can match the historical prefix key-value cache in the host memory, the historical prefix key-value cache is determined as a preliminary pre-filled inference result; Performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a pre-filling reasoning result; Performing recursive reasoning processing according to the pre-filled reasoning result to obtain a recursive reasoning result; The recursive reasoning result is output to the user end.

2. The large model inference method for user requests according to claim 1, characterized in that: The determining whether the request text can match a pre-stored historical prefix key-value cache in the host memory includes: Acquire tree structure data in the host memory, wherein the tree structure data includes one or more tree node data, each tree node data corresponds to a pre-stored historical prefix key-value cache; Get the identification information corresponding to each historical prefix key-value cache; Determine whether the request text can match the corresponding identification information; If the request text can match the corresponding identification information, it is determined that the request text can match the historical prefix key-value cache in the host memory; if the request text cannot match the corresponding identification information, it is determined that the request text cannot match the historical prefix key-value cache in the host memory.

3. The large model inference method for user requests according to claim 1, characterized in that: The performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain the pre-filling reasoning result includes: Performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a prefix key-value cache and an initial word unit; The prefix key-value cache and the initial word-gram are determined as pre-filled inference results.

4. The large model inference method for user requests according to claim 3, characterized in that: After performing pre-filling reasoning processing according to the request text, the preliminary pre-filling reasoning result and the target sentence to obtain a prefix key-value cache and an initial word unit, the method further includes: Decoding the initial word element to obtain a corresponding character string; The character string is output to the client.

5. The large model inference method for user requests according to claim 3, characterized in that: The performing recursive reasoning processing according to the pre-filled reasoning result to obtain the recursive reasoning result includes: Continue to reason based on the initial word, recursively reasoning until a preset end mark is inferred; Get multiple tokens before the last token; Each word-gram is determined as the recursive reasoning result.

6. The large model inference method for user requests according to claim 1, characterized in that: The storage process of the historical prefix key-value cache includes: Get multiple historical prefix key-value caches; The plurality of historical prefix key-value caches are stored in host memory.

7. The large model inference method for user requests according to claim 6, characterized in that: The step of caching the plurality of historical prefix key-value pairs in a host memory comprises: Label each historical prefix matrix data according to a preset data format to obtain each labeled historical prefix key-value cache; Generate tree structure data according to the historical prefix key-value cache after each annotation; The tree structure data is stored in the host memory.

8. The large model inference method for user requests according to claim 1, characterized in that: After determining whether the request text can match the pre-stored historical prefix key-value cache in the host memory, the method further includes: If the request text cannot be matched to the historical prefix key-value cache in the host memory, the request text and the target sentence are input into the preset language model for pre-filled reasoning and recursive reasoning to obtain a corresponding reasoning result; The inference processing result is output to the user end.

9. The large model inference method for user requests according to claim 8, characterized in that: After inputting the request text and the target sentence into a preset language model for pre-filling reasoning and recursive reasoning to obtain corresponding reasoning results, the method further includes: From the inference result, the prefix key-value cache corresponding to the request text is integrated into the tree structure data.

10. The large model inference method for user requests according to claim 1, characterized in that: After determining the historical prefix key-value cache as a preliminary pre-filled inference result if the request text can match the historical prefix key-value cache in the host memory, the method further includes: Obtaining a usage threshold of the host memory; Counting the historical prefix key-value cache to obtain a current count; In the host memory, obtaining a plurality of historical counts; Determine whether the total count quantity of the multiple historical counts and the current count is greater than the usage threshold; If it is determined that the total count quantity is greater than the usage threshold, determining a target tree node to be filtered out in the tree structure data; The target tree node is filtered out from the host memory to update the host memory.

11. The large model inference method for user requests according to claim 1, characterized in that: The process of creating the knowledge base includes: Receiving one or more documents sent by the user; A knowledge base is created based on the one or more documents.

12. The large model inference method for user requests according to claim 11, characterized in that: Determining a corresponding target sentence from a pre-created knowledge base according to the request vector includes: Determining a target document from the knowledge base according to the request vector; According to the request vector, a corresponding target sentence is determined from the target document.

13. A server, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the large model inference method for user requests as claimed in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the large model inference method for user requests as claimed in any one of claims 1 to 12.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model inference method for user request as claimed in any one of claims 1 to 12 are implemented.

Images