Model inference method and related apparatus

Abstract

A model inference method and a related apparatus, which are used for obtaining a pseudo sequence and an attention mask from a plurality of retrieval databases on the basis of an input text, and verifying the pseudo sequence on the basis of a fuzzy verification condition to obtain an inference result, thereby accelerating the inference process of a large language model. The method comprises: on the basis of a plurality of pieces of text data, constructing a plurality of retrieval databases, the text data being obtained by performing inference on a first input text by means of a large language model, and the similarity between the text data in the retrieval databases being higher than a preset value; and, on the basis of the plurality of retrieval databases, performing inference on a second input text by means of the large language model, so as to obtain an inference result of the large language model.

Description

A model inference method and related apparatus

[0001] The present application claims priority to the Chinese patent application No. 202411089074.7, filed on August 8, 2024, and entitled "A model inference method and related apparatus", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a model inference method and related apparatus. BACKGROUND

[0003] With the rapid development of large language models (LLM), its rich open knowledge and powerful reasoning ability are changing the status quo of many technical fields. Especially in the field of recommendation systems, the combination of LLM and recommendation technology has become a hot topic of current research.

[0004] Currently, the industry has proposed to use LLM offline as a recommendation knowledge enhancer to generate data with semantic openness characteristics, such as generating portraits for users and generating additional knowledge for items, and then adapting these feature data to existing traditional recommendation systems. However, in the recommendation scenario, there are massive users and items, and in the process of calling LLM to generate feature data according to a large amount of user and item data, LLM needs to be called multiple times, and the time delay of a single call to LLM is long, resulting in a long time to generate feature data. Therefore, to accelerate the generation of new feature data by LLM, the speculation decoding technology can be combined to accelerate the inference of LLM.

[0005] However, due to the scale of items and users in the recommendation system may be in the order of billions, the current speculation decoding technology obtains a large amount of data from the retrieval pool according to the entire historical knowledge of users and items, which further leads to a very long time to retrieve the draft text according to the clue pool, affecting the inference acceleration of the large language model. SUMMARY

[0006] The present application provides a model inference method and related apparatus for obtaining pseudo-sequences and attention masks from multiple retrieval databases according to input text, and obtaining inference results by verifying the pseudo-sequences based on a fuzzy verification condition, which realizes the acceleration of the inference process of the large language model.

[0007] In a first aspect, the present application provides a model inference method, which comprises: constructing a plurality of retrieval databases according to a plurality of text data. The text data is obtained by inferring a first input text by a large language model, i.e., the text data is historical text data generated by the large language model, and the similarity between the text data in the constructed retrieval database is higher than a first preset value. Subsequently, the second input text can be inferred by the large language model according to the plurality of retrieval databases to obtain the inference result of the large language model.

[0008] In the embodiments of the present application, a plurality of retrieval databases can be constructed according to historical text data, and the text data in the retrieval database has a high similarity, so that subsequent retrieval can be performed from the corresponding retrieval database according to the input text, thereby reducing the amount of data to be retrieved and reducing the retrieval time. In addition, since the text data in each retrieval database has a high similarity, the probability of retrieving similar text in the retrieval database can be improved, thereby reducing the retrieval time.

[0009] In a possible implementation, the foregoing constructing a plurality of retrieval databases according to a plurality of text data can comprise: dividing the plurality of text data according to the similarity of the plurality of text data to obtain a plurality of groups of data, the similarity between the text data in each group being higher than a second preset value; and constructing a plurality of retrieval databases according to the plurality of groups of data, one group of data corresponding to one retrieval database.

[0010] In the embodiments of the present application, the retrieval database is constructed according to the similarity between the text data, which can divide the text data with high similarity into the same retrieval database, thereby improving the probability of retrieving similar data in the retrieval database when retrieving from the corresponding retrieval database according to the input text.

[0011] In a possible implementation, the foregoing inferring the second input text by the large language model according to the plurality of retrieval databases to obtain the inference result of the large language model can comprise: analyzing the plurality of retrieval databases according to the second input text to obtain a pseudo sequence and an attention mask, the pseudo sequence comprising characters of a plurality of draft texts, the draft text being a candidate text of the inference result; and analyzing the second input text, the pseudo sequence and the attention mask by the large language model according to a fuzzy verification condition to obtain the inference result, the fuzzy verification condition being used to determine whether to accept the character in the pseudo sequence according to the conditional probability of the character in the vocabulary at a plurality of character positions, the character position being a position corresponding to the character obtained according to the second input text and the pseudo sequence.

[0012] In the embodiments of the present application, the pseudo sequence can be obtained from the obtained search database, the pseudo sequence can represent a plurality of draft texts obtained based on the input text, so that the subsequent large language model can verify the plurality of draft texts at one time, the verification of the large language model can be accelerated, and then the inference of the large language model can be accelerated.

[0013] In a possible implementation, the foregoing analyzing, according to the second input text, the plurality of search databases to obtain the pseudo sequence and the attention mask can include: determining, according to the second input text, a sub-tree from a target search database, the sub-tree being used to represent a plurality of draft texts, the target search database being determined from the plurality of search databases according to the second input text; analyzing the sub-tree by using a depth-first search algorithm to obtain the pseudo sequence and the attention mask.

[0014] In a possible implementation, the foregoing analyzing, according to the fuzzy verification condition, the second input text, the pseudo sequence and the attention mask by using the large language model to obtain the inference result can include: analyzing, by using the large language model, the second input text, the pseudo sequence and the attention mask to obtain a conditional probability of a character in a character list at a plurality of character positions; verifying, according to the fuzzy verification condition and the conditional probability, the pseudo sequence to obtain the inference result.

[0015] In a possible implementation, the foregoing verifying, according to the fuzzy verification condition and the conditional probability, the pseudo sequence to obtain the inference result can include: when a first character is any one of k second characters, taking the first character as a candidate character of the inference result, the first character being a character corresponding to an i-th character position in the pseudo sequence, the k second characters being characters corresponding to the top k conditional probabilities of the character list at the i-th character position, i being a positive integer from 1 to m, m being a number of characters in the pseudo sequence, and k being a positive integer; determining, according to the candidate character, the inference result.

[0016] In the embodiments of the present application, in combination with the characteristics of the recommendation system, the characters in the draft text can be verified by using the fuzzy verification, the acceptance rate of the large language model for the characters in the draft text can be improved, the hit rate of the draft text is improved, the probability that the large language model accepts multiple characters at one time is increased, and then the inference of the large language model is accelerated.

[0017] In a second aspect, the present application provides a model inference device, comprising: a construction module configured to construct a plurality of search databases according to a plurality of text data, the text data being obtained by inferring a first input text by using a large language model, and the similarity between the text data in the search database being higher than a first preset value; and an inference module configured to infer a second input text by using the large language model according to the plurality of search databases to obtain an inference result of the large language model.

[0018] In a possible implementation, the construction module is specifically configured to: divide the plurality of text data according to similarities of the plurality of text data to obtain a plurality of groups of data, similarity between text data in each group being higher than a second preset value; and construct a plurality of retrieval databases according to the plurality of groups of data, one group of data corresponding to one retrieval database.

[0019] In a possible implementation, the inference module is specifically configured to: analyze the plurality of retrieval databases according to the second input text to obtain a pseudo sequence and an attention mask, the pseudo sequence including characters of a plurality of draft texts, the draft text being a candidate text of the inference result; and analyze the second input text, the pseudo sequence and the attention mask by using the large language model to obtain the inference result according to a fuzzy verification condition, the fuzzy verification condition being used to determine whether to accept a character in the pseudo sequence according to conditional probabilities of the character in the vocabulary at a plurality of character positions, the character position being a position corresponding to the character according to the second input text and the pseudo sequence.

[0020] In a possible implementation, the inference module is specifically configured to: determine a sub-tree from a target retrieval database according to the second input text, the sub-tree being used to represent the plurality of draft texts, the target retrieval database being determined from the plurality of retrieval databases according to the second input text; and analyze the sub-tree by using a depth-first search algorithm to obtain the pseudo sequence and the attention mask.

[0021] In a possible implementation, the inference module is specifically configured to: analyze the second input text, the pseudo sequence and the attention mask by using the large language model to obtain conditional probabilities of characters in the vocabulary at a plurality of character positions; and verify the pseudo sequence according to the fuzzy verification condition and the conditional probabilities to obtain the inference result.

[0022] In a possible implementation, the inference module is specifically configured to: when a first character is any one of k second characters, take the first character as a candidate character of the inference result, the first character being a character corresponding to an i-th character position in the pseudo sequence, the k second characters being characters corresponding to the highest k conditional probabilities of the characters in the vocabulary at the i-th character position, i being a positive integer from 1 to m, m being a number of characters in the pseudo sequence, and k being a positive integer; and determine the inference result according to the candidate character.

[0023] In a third aspect, the present application provides a model inference device, comprising: a processor, a memory, an input and output device, and a bus; the memory stores computer instructions; the processor executes the computer instructions in the memory; the processor executes the computer instructions in the memory to implement the method of the first aspect or any possible implementation manner of the first aspect.

[0024] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium. The computer readable storage medium stores computer instructions; when the computer instructions run on a computer, the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product. The computer program product includes computer programs or instructions; when the computer programs or instructions run on a computer, the computer executes the method in the first aspect or any possible implementation manner of the first aspect.

[0026] The technical effects brought by the second aspect to the fifth aspect or any possible implementation manner thereof can refer to the technical effects brought by the first aspect or the related possible implementation manner of the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0027] FIG. 1 is a schematic diagram of a system architecture provided by the present application;

[0028] FIG. 2 is a schematic diagram of a model inference method provided by the present application;

[0029] FIG. 3 is a process of obtaining a pseudo sequence and an attention mask according to a prefix tree in a search database;

[0030] FIG. 4 is a schematic diagram of an execution process based on the model inference method;

[0031] FIG. 5 is a schematic diagram of a structure of a model inference device provided by the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0033] The method provided in the application can be applied in an artificial intelligence (AI) scene. AI is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer to perceive an environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. AI is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. The research in the field of AI includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0034] First, the overall workflow of the artificial intelligence system is described, and then the above-mentioned artificial intelligence theme framework is described from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0035] (1) Infrastructure

[0041] The infrastructure provides computing power support for the artificial intelligence system, realizes communication with the outside world, and realizes support through the basic platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. hardware acceleration chips); the basic platform includes distributed computing framework and network related platform guarantee and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the basic platform for calculation.

[0037] (2) Data

[0038] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data relates to graphics, images, speech, text, and also relates to Internet of Things data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature and humidity.

[0039] (3) Data processing

[0040] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, and the like.

[0041] Among them, machine learning and deep learning can model, extract, preprocess, train, and the like of symbolic and formalized intelligent information on data.

[0042] Reasoning refers to the process of simulating human intelligent reasoning methods in a computer or intelligent system, using formalized information to solve problems according to reasoning control strategies, and the typical function is search and matching.

[0043] Decision-making refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0044] (4) General capabilities

[0045] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, and the like.

[0046] (5) Intelligent products and industry applications

[0047] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are encapsulations of overall solutions of artificial intelligence, and realize application landing by productizing intelligent information decision-making. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, autonomous driving, smart cities, and the like.

[0048] In order to better understand the scheme of the embodiments of the present application, first, the related terms and concepts that may be involved in the embodiments of the present application are introduced.

[0049] (1) Pre-trained language model

[0050] The pre-trained language model refers to unsupervised learning on a large-scale corpus to obtain word or subword-level representations, and further train specific natural language processing tasks on this basis. The pre-trained language model is usually modeled using a neural network, which can output the probability distribution of the next word or subword in the text by inputting a piece of text. The pre-trained language model has been widely used in the field of natural language processing, including machine translation, speech recognition, text classification, information retrieval, and the like, and has achieved significant improvement in effect.

[0051] (2) Large language model (Large Language Model, LLM)

[0052] LLM refers to a language model containing hundreds of millions (or more) of parameters, which are trained on a large amount of text data, and is a deep learning-based natural language processing model. These models can process large amounts of text data, learning the grammar and semantic rules of natural language. LLM can be applied to text generation, machine translation, question and answer systems, text summarization, or sentiment analysis, etc., and has the advantages of strong generation ability, strong adaptability, accurate prediction, and strong scalability. For example, in the movie recommendation scenario, a large language model can generate a description of the movie scene, including movie type, lead actor, plot, etc., so that the system can better recommend similar movies. A large language model can also generate a recommendation reason, such as an e-commerce website can generate a reason for recommending a product through a large language model, such as the quality, price, and functionality of the product, so that users can better understand the value of the product.

[0053] (3) Traditional recommendation model

[0054] A traditional recommendation model based on collaborative information is an algorithm that uses user behavior data, item attributes, and other information to calculate similarity and establish a user-item relationship model, thereby achieving efficient personalized recommendation. The core idea is to use similar users and items to predict the interest of users in unknown items, thereby providing more accurate and personalized recommendations to users.

[0055] (4) Prefix tree (Trie)

[0056] A prefix tree (also known as a dictionary tree, word search tree, or key tree) is a tree-shaped data structure used to store a collection of strings for fast retrieval, insertion, and deletion operations. Each node in the tree represents a character in a string. The characters along the path from the root node to a certain node are concatenated to form the string represented by that node.

[0057] (5) Depth-first search (Depth-First Search, DFS)

[0058] Depth-first search is an algorithm for traversing or searching trees or graphs. This algorithm traverses the nodes of a tree along its depth, searching as deeply as possible along each branch of the tree. When all the edges out of a node v have been explored, the search backtracks to the node from which v was discovered. This process continues until all nodes reachable from the source node have been discovered. If there are still undiscovered nodes, one of them is selected as the source node and the process is repeated until all nodes have been visited.

[0059] For the inference acceleration of large language models, existing solutions propose speculative decoding technology, which generates multiple candidate sequences in parallel through an auxiliary model (non-autoregressive model or database retrieval), and then verifies the multiple candidate sequences through a large language model, and takes the sequence consistent with the large language model as the inference result, and obtains multiple characters in one decoding. Although this technology effectively shortens the generation time of the large language model through parallelization and early screening, however, for the recommendation system field, facing the knowledge data of the order of hundreds of millions, the database constructed by the existing solution is very large, which leads to a very long time for retrieving multiple candidate sequences, thereby affecting the model inference acceleration effect. In addition, for the recommendation scenario, the tolerance of text diversity generated by the large language model is high, and strict verification is not required.

[0060] To solve the current problem, the embodiment of the present application proposes a model inference method, which can construct multiple retrieval databases according to the similarity between historical knowledge data, so that the data in each retrieval database has high similarity, to balance the retrieval time consumption and the large model decoding time consumption. Subsequently, the multiple draft texts (candidate sequences) obtained based on the multiple retrieval databases can be fuzzy verified based on the fuzzy verification condition, to improve the acceptance rate of the large language model for the characters in the multiple draft texts, and thereby improve the inference speed of the large language model.

[0061] The model inference method proposed by the present application can be applied to various scenarios of information retrieval, such as recommendation systems, search engines, and advertising platforms, and other information retrieval systems with deep neural network models. It can also be applied to each stage of the information retrieval system, such as recall, sorting, and rearrangement.

[0062] In order to make the technical solutions of the present application clearer and easier to understand, the system architecture of the present application will be introduced below in conjunction with the drawings.

[0063] Referring to FIG. 1, a system architecture 100 provided by the present application. The system architecture 100 can include a computing device cluster 110, which includes at least one computing device. Wherein the computing device can be a server, such as a cloud server, a central server, an edge server, or a local server of a local data center. In some possible embodiments, the computing device can also be a desktop computer, a notebook computer, or other terminal device.

[0064] Specifically, the processor used to execute the model inference method provided in the embodiments of the present application can be distributed in any of the computing devices in the computing device cluster 110. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a Ascend processor, etc., and the specific processor is not limited here.

[0065] It is worth noting that the system architecture shown in FIG. 1 is only an example and does not limit the specific implementation of the system architecture to this example. For example, in other possible system architectures, the system architecture 100 can also include a client that sends input text to the computing device, and the computing device obtains inference results according to the input text.

[0066] Next, the method flow provided by the present application will be introduced in combination with the foregoing system architecture.

[0067] Referring to FIG. 2, a flowchart of a model inference method provided by the present application is shown as follows.

[0068] Step 201, constructing a plurality of retrieval databases according to a plurality of text data;

[0069] The text data is historical knowledge data obtained by a large language model performing inference on a first input text, and the text data can be referred to as historical knowledge data. In the embodiments of the present application, the historical knowledge data generated by the large language model can be converted into a plurality of retrieval databases, and each retrieval database includes historical knowledge data with a similarity higher than a first preset value.

[0070] Optionally, the plurality of text data can be divided according to the similarity between the plurality of text data to obtain a plurality of groups of data, and the similarity between the data in each group is higher than a second preset value. After grouping the plurality of text data, a plurality of retrieval databases can be constructed according to the obtained plurality of groups of data, wherein one group of data corresponds to one retrieval database, so that the data in each retrieval database has a high similarity, and when determining the possible draft text from the retrieval database according to the second input text, the relevance between the characters in the draft text can be improved.

[0071] The text data in the retrieval database can be in the form of a prefix tree, and each node of the prefix tree represents a character. A path from a root node to a leaf node is a branch, that is, a piece of text data. The construction of the retrieval database can be performed in various ways, for example, based on clustering similarity, based on attribute characteristics, or based on artificial matching rules, and the like. The specific construction method is not limited here, and different construction methods can be used according to the type of text data.

[0072] Optionally, for the behavior preference knowledge of the user, the user can be clustered to obtain different user groups, and then the historical knowledge data corresponding to the same group of users is divided into the same retrieval database according to the users in different groups. Since the behaviors of the users in the same group are similar, the historical knowledge data corresponding to the same group of users can be made similar, and the data in the retrieval database can have high similarity. In addition, since the behavior of the user can change continuously, the behavior preference knowledge of the user also needs to be updated continuously, and the retrieval database constructed based on the behavior preference knowledge of the user also needs to be updated continuously.

[0073] Optionally, for the basic knowledge of the item, the item can be divided based on the attribute characteristics of the item, and items with the same attribute characteristics are divided into the same category, and then a retrieval database can be constructed according to the knowledge data corresponding to the items in the same category, so as to ensure the similarity of the data in the retrieval database.

[0074] After the construction of the retrieval database according to the historical knowledge data is completed, the prefix tree in the retrieval database can be updated according to new knowledge data generated by the large language model, and the new knowledge data is added to the prefix tree for dynamic adjustment. The new knowledge data is text data obtained by inputting the historical knowledge data and the second input text into the large language model.

[0075] In the embodiments of the present application, a plurality of retrieval databases can be constructed according to the similarity between the text data, that is, the historical knowledge data. The data in each retrieval database has high similarity. By converting the historical knowledge data into a plurality of retrieval databases, the subsequent retrieval according to the input text can select the corresponding retrieval database for retrieval, thereby reducing the amount of data for retrieval, and the data in each retrieval database has high similarity, which can improve the probability of retrieving similar text from the retrieval database, thereby reducing the time consumption of retrieval and the time consumption of large model decoding.

[0076] Step 202, according to the plurality of retrieval databases, inferring the second input text by the large language model to obtain the inference result of the large language model.

[0077] After the plurality of retrieval databases are constructed, based on the speculative decoding technique, a plurality of draft texts can be determined from the retrieval databases according to the second input text s = {x1, x2, …, x t The draft texts are possible inference results of the large language model for the second input text, i.e., the candidate texts of the inference results.

[0078] Optionally, a target retrieval database can be determined from the plurality of retrieval databases according to the second input text, and then the last N characters of the second input text can be taken as a prefix, where N is a positive integer. According to the prefix, a subtree can be determined from a prefix tree of the target retrieval database, the subtree is also a prefix tree, and the subtree includes a plurality of draft texts. Each branch in the subtree can represent a draft text.

[0079] Optionally, after the subtree is obtained, the subtree can be analyzed by a depth-first search algorithm to obtain a pseudo sequence and a tree attention mask, or the subtree can be analyzed by a prefix search, or other retrieval algorithms, which are not limited here. The pseudo sequence includes characters of the plurality of draft texts, and the pseudo sequence can be represented as The attention mask is used to obtain the draft texts according to the pseudo sequence.

[0080] As shown in FIG. 3, exemplarily, the second input text is a cat sits, s = {a, cat, sits}, and sits is taken as a prefix. A subtree can be determined from a prefix tree in one of the retrieval databases, and it can be known from the subtree that the subsequent possible draft texts can be on my knee, on a table, or on a chair. According to the determined subtree, the pseudo sequence can be {on, my, knee, a, table, chair}, and in addition, the tree attention mask shown in FIG. 3, the black part represents the characters shown in the pseudo sequence, and the white part represents the characters in the pseudo sequence to be masked. Therefore, according to the pseudo sequence and the attention mask, the plurality of draft texts can be represented.

[0081] Optionally, after the pseudo sequence and the attention mask are obtained, the inference result can be obtained by analyzing the second input text, the pseudo sequence, and the attention mask by the large language model according to a fuzzy verification condition. The fuzzy verification condition is used to determine whether to accept the characters in the pseudo sequence according to the conditional probability of the characters in the vocabulary at a plurality of character positions.

[0082] Specifically, the second input text, the pseudo sequence and the attention mask can be analyzed by the large language model to obtain conditional probabilities of all characters in the character set at multiple character positions, which are positions corresponding to the characters obtained according to the second input text and the pseudo sequence. When the pseudo sequence includes m characters, there are m character positions. The conditional probability of a character in the character set at the i th character position is the probability of outputting the character in the character set at the i th character position based on the current context. The calculation formula of the conditional probability of a character in the character set at the i th character position satisfies where i is a positive integer from 1 to m, x represents any character in the character set, and the current context at this time includes multiple characters. Subsequently, the pseudo sequence can be verified based on the fuzzy verification condition and the conditional probabilities of all characters in the character set at the multiple character positions to obtain the inference result.

[0083] In the embodiments of the present application, the pseudo sequence is verified based on the fuzzy verification condition and the conditional probability, that is, multiple draft texts can be verified at one time, and through parallel verification processing, the generation time of the large language model can be shortened.

[0084] Generally, for the draft text obtained based on speculative decoding, a strict verification condition is performed, that is, the character with the highest conditional probability at the current position is the same as the character at the corresponding position in the draft text, and only then can the text at the current position of the draft text be accepted. However, since the recommended model does not require completely accurate matching text as input, in the embodiments of the present application, the characters corresponding to the top k highest conditional probabilities before the current position can be used to verify the characters at the corresponding positions in the pseudo sequence, so that the large language model can accept more characters, improve the character acceptance rate, and the probability condition can effectively avoid divergent generation, that is, the length of the generated text can be controlled to avoid generating too long text. Wherein, k can be a number between 3 and 5, and the specific value is not limited. Moreover, when the predecessor node of the current node in the draft text is not accepted, the current node does not need to be verified, which can reduce the verification time and further speed up the inference of the large language model.

[0085] Optionally, when the first character is any one of the k second characters, the first character can be taken as a candidate character of the inference result, wherein the first character is the character corresponding to the i th position in the pseudo sequence, and the k second characters are the characters corresponding to the top k conditional probabilities at the i th character position in the character set. After verifying the pseudo sequence to obtain multiple candidate characters, the longest branch composed of the candidate characters can be taken as the inference result and output according to the ordering of the candidate characters in the sub-tree, or other branches can be selected as the inference result, that is, the longest branch can not be selected, and the specific place is not limited.

[0086] The conditional probability of each character in the vocabulary at the i-th character position can be calculated, then the plurality of conditional probabilities are sorted from high to low, and the k second characters corresponding to the top k conditional probabilities can be obtained. Optionally, the k characters corresponding to the top k conditional probabilities can be further screened by judging whether the conditional probability of the k characters is greater than a preset probability, and the k characters are further screened. If the conditional probability is less than the preset probability, the character cannot be used as a second character. The number of obtained second characters is less than or equal to k.

[0087] Optionally, when the first character is not any of the k second characters, the character with the highest conditional probability in the vocabulary can be used as the character at the current position.

[0088] For example, for the pseudo sequence {on, my, knee, a, table, chair}, the pseudo sequence includes 6 characters. Therefore, first, the conditional probability of each character in the vocabulary at the first character position can be obtained according to the second input text s = {a, cat, sits}. Then, the character on at the first position in the pseudo sequence can be verified according to the conditional probability. Whether on is any of the k second characters can be determined. If yes, the character on at the first position in the pseudo sequence is accepted. Similarly, if my, knee and a in the pseudo sequence are accepted, and table and chair are not the top k characters corresponding to the corresponding positions, table and chair are not accepted. At this time, {on, my, knee} can be used as the inference result, and three tokens are generated at one time by using the large language model, thereby increasing the proportion of tokens accepted by the large language model and accelerating the inference process of the large model.

[0089] The text knowledge obtained by the model inference method proposed in the present application can be converted into a dense vector by an encoder and used for training of a downstream recommendation model or other model.

[0090] Fig. 4 is a schematic diagram of an execution process of a model inference method according to an embodiment of the present application. As shown in Fig. 4, first, a plurality of retrieval pools can be constructed according to the knowledge data corresponding to the user and the item, and a plurality of draft texts can be obtained by retrieving the input text through the constructed plurality of retrieval pools. For example, the input text is the user, and the obtained draft text is loves thriller films. Subsequently, a fuzzy verification can be performed, and the draft text can be verified by a large language model. In the first character position, the conditional probabilities of all characters in the first position in the word table can be sorted from high to low, and the top two conditional probabilities corresponding to the characters can be selected to verify the characters in the draft text. As shown in Fig. 4, the conditional probabilities of like and loves in the word table are the top two among the conditional probabilities of all characters, and the loves in the draft text belongs to one of like and loves, so the character can be accepted. Finally, the three characters of loves thriller films can be output.

[0091] Exemplarily, based on the Movielens-10M and Amazon-Books two datasets, Vanilla, single pool, random pool, customized pool (CP), relaxed verification (RV), and CP+RV (the model inference method proposed in the present application) are used respectively to compare and analyze the average accepted token, speed, retrieval time, total time, and speed up from five aspects. The comparison results are shown in Table 1.

[0092] Table 1

[0093] As shown in Table 1, the present application achieves a speed up of x4.52 and x4.75 on the ML-10M and Amazon-books datasets respectively, and the acceleration effect is significant. The retrieval database constructed in the present application greatly reduces the retrieval time, and the fuzzy verification proposed in the present application can significantly increase the number of accepted tokens. In addition, since the fuzzy verification will produce a text different from the original model, the effects of the knowledge generated by the two strategies of accurate verification and fuzzy verification on the downstream task are compared, and the comparison results are shown in Table 2. As shown in Table 2, the fuzzy verification does not affect the effect of the downstream task.

[0094] Table 2

[0095] The foregoing introduces the method provided by the present application, and the following introduces the device provided by the present application based on the foregoing method.

[0096] Referring to FIG. 5, a structural schematic diagram of a model reasoning device provided by the present application is shown as follows.

[0097] The construction module 501 is configured to construct a plurality of retrieval databases according to a plurality of text data, the text data being obtained by reasoning a first input text by a large language model, and a similarity between the text data in the retrieval database being higher than a first preset value;

[0098] The reasoning module 502 is configured to reason a second input text by the large language model according to the plurality of retrieval databases to obtain a reasoning result of the large language model.

[0099] In a possible implementation, the foregoing construction module 501 is specifically configured to: divide the plurality of text data according to the similarity of the plurality of text data to obtain a plurality of groups of data, a similarity between text data in each group being higher than a second preset value; and construct the plurality of retrieval databases according to the plurality of groups of data, one group of data corresponding to one retrieval database.

[0100] In a possible implementation, the foregoing reasoning module 502 is specifically configured to: analyze the plurality of retrieval databases according to the second input text to obtain a pseudo sequence and an attention mask, the pseudo sequence including characters of a plurality of draft texts, the draft text being a candidate text of the reasoning result; and analyze the second input text, the pseudo sequence and the attention mask by the large language model according to a fuzzy verification condition to obtain the reasoning result, the fuzzy verification condition being used to determine whether to accept a character in the pseudo sequence according to a conditional probability of the character in a character position in a vocabulary, the character position being a position corresponding to the character obtained according to the second input text and the pseudo sequence.

[0101] In a possible implementation, the foregoing reasoning module 502 is specifically configured to: determine a sub-tree from a target retrieval database according to the second input text, the sub-tree being used to represent the plurality of draft texts, the target retrieval database being determined from the plurality of retrieval databases according to the second input text; and analyze the sub-tree by a depth-first search algorithm to obtain the pseudo sequence and the attention mask.

[0102] In a possible implementation, the foregoing reasoning module 502 is specifically configured to: analyze the second input text, the pseudo sequence and the attention mask by the large language model to obtain a conditional probability of a character in a character position in a vocabulary; and verify the pseudo sequence according to the fuzzy verification condition and the conditional probability to obtain the reasoning result.

[0103] In a possible implementation, the foregoing reasoning module 502 is specifically configured to: when the first character is any one of k second characters, taking the first character as a candidate character of the reasoning result, the first character being a character corresponding to an i-th character position in the pseudo sequence, the k second characters being characters corresponding to top k conditional probabilities of the i-th character position in the word table, i being a positive integer from 1 to m, m being a number of characters in the pseudo sequence, and k being a positive integer; and determining the reasoning result according to the candidate character.

[0104] The embodiment of the present application further provides a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute the method provided by the present application.

[0105] The embodiment of the present application further provides a computer program product containing instructions. The computer program product can be software or a program product containing instructions, which can run on a computing device or be stored in any available medium. When the computer program product runs on at least one computing device, the at least one computing device is caused to execute the method provided by the present application.

[0106] In addition, it should be noted that the apparatus embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or distributed on a plurality of network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiments provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0107] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, apparatus and unit can refer to the corresponding process in the foregoing method embodiments, which will not be described here.

[0108] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is only a logical function division. There can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different units, can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

[0109] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to the actual needs to achieve the purposes of the embodiments of the present application.

[0110] In addition, each functional unit in the embodiments of the present application can be integrated in one processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of software functional units.

[0111] When the integrated unit is implemented in the form of software functional units and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), magnetic disk or optical disk, and various other media that can store program codes.

[0112] The terms "first", "second", "third", "fourth", and the like in the description and in the claims of the present application, and above-mentioned drawings, if any, are used as identifiers for distinguishing between similar objects, and do not necessarily describe a particular chronological or sequential order. It will be understood that the use of such terms is arbitrary and made solely for the purpose of distinguishing between the similar objects. It is to be understood that data so used in the specification can be interchanged, under suitable circumstances, to convey a given embodiment described herein in an alternate order. Furthermore, the terms "comprising" and "including" and any of their derivatives, are intended to be construed as encompassing a non-exclusive inclusion, e.g., a process, method, system, product or apparatus that includes a list of steps or units as limitations can not necessarily be limited to the steps or units that are expressly listed, but can include other steps or units that are not expressly listed or inherent to such process, method, product or apparatus.

[0113] Finally, it should be noted that the above only is a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A model inference method, comprising: The method comprises the following steps: According to a plurality of text data, a plurality of retrieval databases are constructed, wherein the text data is obtained by reasoning a first input text through a large language model, and the similarity between the text data in the retrieval database is higher than a first preset value; According to the plurality of retrieval databases, a second input text is reasoned through the large language model to obtain a reasoning result of the large language model.

2. The method of claim 1, wherein, According to a plurality of text data, a plurality of retrieval databases are constructed, comprising: According to the similarity of the plurality of text data, the plurality of text data is divided to obtain a plurality of groups of data, and the similarity between the text data in each group is higher than a second preset value; According to the plurality of groups of data, the plurality of retrieval databases are constructed, and one group of data corresponds to one retrieval database.

3. The method according to claim 1 or 2, characterized in that, According to a plurality of retrieval databases, a second input text is reasoned through the large language model to obtain a reasoning result of the large language model, comprising: According to the second input text, the plurality of retrieval databases are analyzed to obtain a pseudo sequence and an attention mask, wherein the pseudo sequence comprises a plurality of draft text characters, and the draft text is a candidate text of the reasoning result; According to a fuzzy verification condition, the second input text, the pseudo sequence and the attention mask are analyzed through the large language model to obtain the reasoning result, wherein the fuzzy verification condition is used to determine whether to accept the character in the pseudo sequence according to the conditional probability of the character in the character table at a plurality of character positions, and the character position is the position corresponding to the character obtained according to the second input text and the pseudo sequence.

4. The method of claim 3, wherein, According to the second input text, the plurality of retrieval databases are analyzed to obtain a pseudo sequence and an attention mask, comprising: According to the second input text, a sub-tree is determined from a target retrieval database, wherein the sub-tree is used to represent the plurality of draft texts, and the target retrieval database is determined from the plurality of retrieval databases according to the second input text; The sub-tree is analyzed through the depth-first search algorithm to obtain the pseudo sequence and the attention mask.

5. The method according to claim 3 or 4, characterized in that, According to a fuzzy verification condition, the second input text, the pseudo sequence and the attention mask are analyzed through the large language model to obtain the reasoning result, comprising: The conditional probability of the character in the character table at the plurality of character positions is obtained by analyzing the second input text, the pseudo sequence and the attention mask through the large language model; According to the fuzzy verification condition and the conditional probability, the pseudo sequence is verified to obtain the reasoning result.

6. The method of claim 5, wherein, According to the fuzzy verification condition and the conditional probability, the pseudo sequence is verified to obtain the reasoning result, comprising: when a first character is any one of k second characters, the first character is a character corresponding to an i-th character position in the pseudo sequence, the k second characters are characters corresponding to top k conditional probabilities at the i-th character position in the vocabulary, i is a positive integer from 1 to m, m is a number of characters in the pseudo sequence, and k is a positive integer; determine the inference result according to the candidate character.

7. A model inference apparatus characterized by comprising: Comprise: The construction module is used for constructing a plurality of retrieval databases according to a plurality of text data, the text data being obtained by reasoning a first input text through a large language model, and similarity between text data in the retrieval database being higher than a first preset value; The inference module is used for reasoning a second input text through the large language model according to the plurality of retrieval databases to obtain an inference result of the large language model.

8. The apparatus of claim 7, wherein, The construction module is specifically used for: dividing the plurality of text data according to similarity of the plurality of text data to obtain a plurality of groups of data, similarity between text data in each group being higher than a second preset value; constructing the plurality of retrieval databases according to the plurality of groups of data, one group of data corresponding to one retrieval database.

9. The apparatus of claim 7 or 8, wherein, The inference module is specifically used for: analyzing the plurality of retrieval databases according to the second input text to obtain a pseudo sequence and an attention mask, the pseudo sequence comprising characters of a plurality of draft texts, the draft text being a candidate text of the inference result; analyzing the second input text, the pseudo sequence and the attention mask through the large language model according to a fuzzy verification condition to obtain the inference result, the fuzzy verification condition being used for judging whether to accept a character in the pseudo sequence according to a conditional probability of the character at a plurality of character positions in a vocabulary, the character position being a position corresponding to the character obtained according to the second input text and the pseudo sequence.

10. The apparatus of claim 9, wherein, The inference module is specifically used for: determining a sub-tree from a target retrieval database according to the second input text, the sub-tree being used for representing the plurality of draft texts, the target retrieval database being determined from the plurality of retrieval databases according to the second input text; analyzing the sub-tree through the depth-first search algorithm to obtain the pseudo sequence and the attention mask.

11. The apparatus of claim 9 or 10, wherein, The inference module is specifically used for: analyzing the second input text, the pseudo sequence and the attention mask through the large language model to obtain a conditional probability of a character in the vocabulary at the plurality of character positions; verifying the pseudo sequence according to the fuzzy verification condition and the conditional probability to obtain the inference result.

12. The apparatus of claim 11, wherein, The inference module is specifically used for: when the first character is any one of k second characters, the first character being a character corresponding to an i-th character position in the pseudo sequence, the k second characters being characters corresponding to top k conditional probabilities at the i-th character position in the vocabulary, i being a positive integer from 1 to m, m being a number of characters in the pseudo sequence, and k being a positive integer, the first character is taken as a candidate character of the inference result; determining the inference result according to the candidate character.

13. A model inference apparatus comprising: comprise: a processor and a memory, the processor being coupled to the memory; the memory, configured to store a program; the processor, configured to execute the program in the memory, so as to perform the method in any one of claims 1 to 6.

14. A computer readable storage medium comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 6.

15. A computer program product comprising instructions which, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 6.

Images