Model reasoning method and related device

Abstract

The embodiment of the invention provides a model reasoning method and a related device, which are used for obtaining a pseudo sequence and an attention mask from a plurality of retrieval databases according to an input text, and verifying the pseudo sequence based on a fuzzy verification condition to obtain a reasoning result, so that the acceleration of a large language model reasoning process is realized. The method comprises the steps that multiple retrieval databases are constructed according to multiple pieces of text data, the text data are obtained by reasoning a first input text through a large language model, and the similarity between the text data in the retrieval databases is higher than a preset value; and according to the plurality of retrieval databases, reasoning a second input text through the large language model to obtain a reasoning result of the large language model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a model reasoning method and related apparatus. Background Technology

[0002] With the rapid development of Large Language Models (LLMs), their rich open knowledge and powerful reasoning capabilities are changing the landscape of many technological fields. Especially in the field of recommender systems, the combination of LLMs and recommender techniques has become a hot research topic.

[0003] Currently, the industry has proposed using LLM (Limited Learning Model) offline as a recommendation knowledge enhancer to generate data with semantically open features, such as user profiles and additional knowledge for items. This feature data is then adapted to existing traditional recommendation systems. However, in recommendation scenarios involving massive amounts of users and items, generating feature data from a large dataset requires multiple LLM calls, each with a significant latency, resulting in lengthy feature data generation times. Therefore, to accelerate LLM generation of new feature data, speculative decoding techniques can be combined to speed up LLM inference.

[0004] However, since the scale of items and users in a recommendation system can be in the hundreds of millions, current speculative decoding techniques generate a massive amount of data in the retrieval pool based on all historical knowledge of users and items. This results in a very long time to retrieve draft text from this clue pool, which impacts the inference acceleration of large language models. Summary of the Invention

[0005] This application provides a model inference method and related apparatus, which is used to obtain pseudo-sequences and attention masks from multiple retrieval databases based on input text, and obtain inference results by verifying the pseudo-sequences based on fuzzy verification conditions, thereby accelerating the inference process of large language models.

[0006] Firstly, this application provides a model inference method, which includes: constructing multiple retrieval databases based on multiple text data. The text data is obtained by inferring from a first input text using a large language model; that is, the text data is historical text data generated by the large language model, and the similarity between the text data in the constructed retrieval databases is higher than a first preset value. Subsequently, based on the multiple retrieval databases, a second input text can be inferred using the large language model to obtain the inference result of the large language model.

[0007] In this embodiment, multiple retrieval databases can be constructed based on historical text data, and the text data in the retrieval databases can have a high degree of similarity. This allows subsequent searches to be performed from the corresponding retrieval databases based on the input text, reducing the amount of data to be retrieved and thus reducing retrieval time. Furthermore, since the text data in each retrieval database has a high degree of similarity, the probability of finding similar text in the retrieval database can be increased, thereby reducing retrieval time.

[0008] In one possible implementation, the aforementioned construction of multiple retrieval databases based on multiple text data may include: dividing the multiple text data into multiple sets of data based on the similarity of the multiple text data, wherein the similarity between the text data in each set is higher than a second preset value; and constructing multiple retrieval databases based on the multiple sets of data, wherein each set of data corresponds to one retrieval database.

[0009] In this embodiment of the application, a retrieval database is constructed based on the similarity between text data. Text data with high similarity can be grouped into the same retrieval database. When retrieving from the corresponding retrieval database based on the input text, the probability of retrieving similar data in the retrieval database can be increased.

[0010] In one possible implementation, the aforementioned reasoning process based on multiple retrieval databases and using a large language model to obtain the reasoning result of the large language model may include: analyzing multiple retrieval databases based on the second input text to obtain a pseudo-sequence and an attention mask, wherein the pseudo-sequence includes characters from multiple draft texts, and the draft texts are candidate texts for the reasoning result; and analyzing the second input text, the pseudo-sequence, and the attention mask using a large language model based on fuzzy verification conditions to obtain the reasoning result, wherein the fuzzy verification conditions are used to determine whether to accept characters in the pseudo-sequence based on the conditional probabilities of characters in the vocabulary at multiple character positions, and the character positions are the positions corresponding to the characters obtained from the second input text and the pseudo-sequence.

[0011] In this embodiment of the application, a pseudo-sequence can be obtained from the retrieved database. This pseudo-sequence can represent multiple draft texts obtained based on the input text, so that the subsequent large language model can verify multiple draft texts at once, which can accelerate the verification of the large language model and thus accelerate the inference of the large language model.

[0012] In one possible implementation, the aforementioned analysis of multiple retrieval databases based on the second input text to obtain pseudo-sequences and attention masks may include: determining a subtree from a target retrieval database based on the second input text, the subtree representing multiple draft texts, the target retrieval database being determined from multiple retrieval databases based on the second input text; and analyzing the subtree using a depth-first search algorithm to obtain pseudo-sequences and attention masks.

[0013] In one possible implementation, the aforementioned analysis of the second input text, pseudo-sequence, and attention mask using a large language model based on fuzzy verification conditions to obtain the inference result may include: analyzing the second input text, pseudo-sequence, and attention mask using a large language model to obtain the conditional probabilities of characters in the vocabulary at multiple character positions; verifying the pseudo-sequence based on the fuzzy verification conditions and conditional probabilities to obtain the inference result.

[0014] In one possible implementation, the aforementioned verification of the pseudo-sequence based on fuzzy verification conditions and conditional probabilities to obtain the inference result may include: when the first character is any one of the k second characters, the first character is taken as a candidate character for the inference result, where the first character is the character corresponding to the i-th character position in the pseudo-sequence, the k second characters are the characters corresponding to the highest k conditional probabilities at the i-th character position in the vocabulary, i is a positive integer from 1 to m, m is the number of characters in the pseudo-sequence, and k is a positive integer; the inference result is determined based on the candidate character.

[0015] In this embodiment of the application, combined with the characteristics of the recommendation system, fuzzy verification can be used to verify the characters in the draft text, which can improve the acceptance rate of the large language model for the characters in the draft text, improve the hit rate of the draft text, increase the probability of the large language model accepting multiple characters at once, and thus accelerate the reasoning of the large language model.

[0016] Secondly, this application provides a model reasoning device, comprising: a construction module for constructing multiple retrieval databases based on multiple text data, wherein the text data are obtained by reasoning about a first input text through a large language model, and the similarity between the text data in the retrieval databases is higher than a first preset value; and a reasoning module for reasoning about a second input text through a large language model based on the multiple retrieval databases, thereby obtaining the reasoning result of the large language model.

[0017] In one possible implementation, the aforementioned construction module is specifically used to: divide multiple text data into multiple sets of data based on the similarity of the multiple text data, wherein the similarity between the text data in each set is higher than a second preset value; and construct multiple retrieval databases based on the multiple sets of data, wherein each set of data corresponds to one retrieval database.

[0018] In one possible implementation, the aforementioned inference module is specifically used for: analyzing multiple retrieval databases based on the second input text to obtain a pseudo-sequence and an attention mask, wherein the pseudo-sequence includes characters from multiple draft texts, and the draft texts are candidate texts for the inference result; and analyzing the second input text, the pseudo-sequence, and the attention mask using a large language model based on fuzzy verification conditions to obtain the inference result, wherein the fuzzy verification conditions are used to determine whether to accept characters in the pseudo-sequence based on the conditional probabilities of characters in the vocabulary at multiple character positions, and the character positions are the positions corresponding to the characters obtained based on the second input text and the pseudo-sequence.

[0019] In one possible implementation, the aforementioned reasoning module is specifically used to: determine a subtree from a target retrieval database based on the second input text, the subtree representing multiple draft texts, the target retrieval database being determined from multiple retrieval databases based on the second input text; and analyze the subtree using a depth-first search algorithm to obtain pseudo-sequences and attention masks.

[0020] In one possible implementation, the aforementioned inference module is specifically used to: analyze the second input text, pseudo-sequence, and attention mask using a large language model to obtain the conditional probabilities of characters in the vocabulary at multiple character positions; and verify the pseudo-sequence based on the fuzzy verification conditions and conditional probabilities to obtain the inference result.

[0021] In one possible implementation, the aforementioned reasoning module is specifically used to: when the first character is any one of the k second characters, use the first character as a candidate character for the reasoning result, where the first character is the character corresponding to the i-th character position in the pseudo-sequence, the k second characters are the characters corresponding to the highest conditional probabilities at the i-th character position in the vocabulary, i is a positive integer from 1 to m, m is the number of characters in the pseudo-sequence, and k is a positive integer; and determine the reasoning result based on the candidate character.

[0022] Thirdly, this application provides a model inference apparatus, which includes: a processor, a memory, an input / output device, and a bus; the memory stores computer instructions; when the processor executes the computer instructions in the memory, the memory stores computer instructions; when the processor executes the computer instructions in the memory, it is used to implement a method as described in the first aspect or any possible implementation of the first aspect.

[0023] Fourthly, embodiments of this application provide a computer-readable storage medium. The computer-readable storage medium stores computer instructions; when the computer instructions are executed on a computer, the computer causes the computer to perform the method as described in the first aspect or any possible implementation thereof.

[0024] Fifthly, embodiments of this application provide a computer program product. This computer program product includes a computer program or instructions that, when executed on a computer, cause the computer to perform the method described in the first aspect or any possible implementation thereof.

[0025] The technical effects of the second to fifth aspects or any of their possible implementations can be found in the first aspect or the related possible implementations of the first aspect, and will not be repeated here. Attached Figure Description

[0026] Figure 1 A schematic diagram of a system architecture is provided for this application;

[0027] Figure 2 A flowchart illustrating a model reasoning method provided in this application;

[0028] Figure 3 This is the process of obtaining pseudo-sequences and attention masks based on the prefix tree in the retrieved database;

[0029] Figure 4 This is a schematic diagram illustrating the execution process of the model-based reasoning method.

[0030] Figure 5 This is a schematic diagram of the structure of a model reasoning device provided in this application. Detailed Implementation

[0031] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0032] The method provided in this application can be applied to artificial intelligence (AI) scenarios. AI is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence studies the design principles and implementation methods of various intelligent machines, enabling them to have the functions of perception, reasoning, and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, and fundamental AI theories.

[0033] First, the overall workflow of an artificial intelligence system is described. The following sections elaborate on the aforementioned AI framework from two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed by technology) to the industrial ecosystem of the system.

[0034] (1) Infrastructure

[0035] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0036] (2) Data

[0037] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0038] (3) Data processing

[0039] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0040] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0041] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0042] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0043] (4) General ability

[0044] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0045] (5) Smart Products and Industry Applications

[0046] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0047] To better understand the solutions of the embodiments of this application, the relevant terms and concepts that may be involved in the embodiments of this application will be introduced first.

[0048] (1) Pre-trained language model

[0049] Pre-trained language models refer to unsupervised learning on large-scale corpora to obtain word or sub-word representations, which are then further trained for specific natural language processing tasks. Pre-trained language models typically use neural networks for modeling and can output the probability distribution of the next word or sub-word in a given text as input. Pre-trained language models have been widely used in natural language processing, including machine translation, speech recognition, text classification, and information retrieval, achieving significant performance improvements.

[0050] (2) Large Language Model (LLM)

[0051] LLM (Language Modeling) refers to a language model with hundreds of millions (or more) parameters trained on massive amounts of text data. It's a deep learning-based natural language processing model. These models can process large amounts of text data to learn the grammatical and semantic rules of natural language. LLMs can be applied to text generation, machine translation, question answering systems, text summarization, and sentiment analysis, offering advantages such as strong generative capabilities, high adaptability, accurate prediction, and scalability. For example, in movie recommendation scenarios, large language models can generate descriptions of movie scenes, including genre, main actors, and plot, enabling the system to better recommend similar films. Large language models can also generate recommendation reasons; for instance, e-commerce websites can use large language models to generate reasons for recommending products, such as product quality, price, and features, allowing users to better understand the value of the products.

[0052] (3) Traditional recommendation model

[0053] Traditional recommendation models based on collaborative information utilize user behavior data and item attributes to calculate similarity and establish user-item relationship models, thereby achieving efficient personalized recommendations. The core idea is to use similar users and items to predict a user's interest in unknown items, thus providing more accurate and personalized recommendations.

[0054] (4) Trie

[0055] A prefix tree (also known as a trie, word lookup tree, or key tree) is a tree-like data structure used to store collections of strings for fast retrieval, insertion, and deletion operations. Each node in the tree represents a character in the string. The characters on the path from the root node to a given node, when concatenated, form the string represented by that node.

[0056] (5) Depth-First Search (DFS)

[0057] Depth-first search (DFS) is an algorithm for traversing or searching trees or graphs. This algorithm traverses the nodes of a tree along its depth, searching the branches as deeply as possible. When all edges containing node v have been explored, the search backtracks to the starting node of the edge that found node v. This process continues until all nodes reachable from the source node have been found. If any undiscovered nodes exist, one of them is selected as the source node, and the above process is repeated until all nodes have been visited.

[0058] To accelerate inference using large language models, existing solutions have proposed speculative decoding techniques. These techniques generate multiple candidate sequences in parallel using an auxiliary model (non-autoregressive model or database retrieval), then validate these candidate sequences against the large language model, using the sequence consistent with the large language model's generation as the inference result, thus obtaining multiple characters in a single decoding operation. While this technique effectively shortens the generation time of the large language model through parallelization and pre-screening, for recommendation systems dealing with hundreds of millions of pieces of knowledge data, the databases built using existing solutions are extremely large, resulting in very long retrieval times for multiple candidate sequences, thereby impacting the model's inference acceleration performance. Furthermore, recommendation scenarios have a high tolerance for text diversity generated by large language models and do not require rigorous validation.

[0059] To address the current problem, this application proposes a model inference method. This method constructs multiple retrieval databases based on the similarity between historical knowledge data, ensuring high similarity among the data in each database to balance retrieval time and large model decoding time. Subsequently, fuzzy verification is performed on multiple draft texts (candidate sequences) obtained from the multiple retrieval databases based on fuzzy verification conditions. This improves the acceptance rate of characters in the multiple draft texts by the large language model, thereby increasing the inference speed of the large language model.

[0060] The model reasoning method proposed in this application can be applied to various information retrieval scenarios, such as information retrieval systems with deep neural network models, such as recommendation systems, search engines, and advertising platforms. It can also be applied to various stages of information retrieval systems, such as recall, ranking, and reordering.

[0061] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be described below with reference to the accompanying drawings.

[0062] See Figure 1 This application provides a system architecture 100. The system architecture 100 may include a computing device cluster 110, which includes at least one computing device. The computing device may be a server, such as a cloud server, central server, edge server, or local server in a local data center. In some possible embodiments, the computing device may also be a desktop computer, laptop computer, or other terminal device.

[0063] Specifically, the processor used to execute the model inference method proposed in the embodiments of this application can be distributed in any one of the computing devices in the aforementioned computing device cluster 110. The processor can be a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), an Ascend processor, etc., and is not limited here.

[0064] It is worth noting that, Figure 1 The system architecture shown is merely an example and is not intended to limit its specific implementation to this example. For example, in other possible system architectures, system architecture 100 may also include the client that sends input text to the computing device, which then obtains inference results based on the input text.

[0065] The method flow provided in this application will be described below in conjunction with the aforementioned system architecture.

[0066] See Figure 2 The flowchart of a model reasoning method provided in this application is as follows.

[0067] Step 201: Construct multiple retrieval databases based on multiple text data;

[0068] The text data refers to historical knowledge data obtained by the large language model from reasoning about the first input text. In this embodiment, the historical knowledge data generated by the large language model can be converted into multiple retrieval databases, each including historical knowledge data with a similarity higher than a first preset value.

[0069] Optionally, multiple text data can be divided into multiple groups based on their similarity, with the similarity between data in each group being higher than a second preset value. After grouping the multiple text data, multiple retrieval databases can be constructed based on the obtained multiple groups of data, with one retrieval database corresponding to each group of data, so that the data in each retrieval database has a high similarity. When determining possible draft text from the retrieval database based on the second input text, the relevance between characters in the draft text can be improved.

[0070] The text data in the retrieval database can be presented in the form of a prefix tree. Each node in the prefix tree represents a character, and a path from the root node to a leaf node is a branch, i.e., a piece of text data. The retrieval database can be constructed in various ways, such as based on cluster similarity, attribute features, or manual matching rules. No specific method is limited here; different methods can be used depending on the category of the text data.

[0071] Optionally, based on user behavioral preference knowledge, users can be clustered to obtain different user groups. Subsequently, based on users within different groups, the historical knowledge data corresponding to the same group of users can be assigned to the same retrieval database. Since users within the same group exhibit similar behaviors, their corresponding historical knowledge data can also be similar, resulting in high similarity among the data in the retrieval database. Furthermore, because user behavior may continuously change, the user behavioral preference knowledge also needs to be continuously updated, and consequently, the retrieval database built based on user behavioral preference knowledge also needs to be continuously updated.

[0072] Optionally, based on the basic knowledge of items, items can be classified according to their attribute characteristics, and items with the same attribute characteristics can be grouped into the same category. Subsequently, a retrieval database can be constructed based on the knowledge data corresponding to items in the same category, thereby ensuring the similarity of data in the retrieval database.

[0073] After constructing the retrieval database based on historical knowledge data, the prefix tree in the retrieval database can be updated based on new knowledge data generated by the large language model. The new knowledge data is added to the prefix tree, dynamically adjusting it. Here, the new knowledge data is the text data obtained by the large language model based on historical knowledge data and the second input text.

[0074] In this embodiment, multiple retrieval databases can be constructed based on the similarity between text data, i.e., historical knowledge data, with each database having a high degree of similarity. By transforming historical knowledge data into multiple retrieval databases, subsequent retrieval based on the input text can select the corresponding database, thereby reducing the amount of data retrieved. Furthermore, the high similarity between the data in each retrieval database increases the probability of retrieving similar text, thus reducing retrieval time and the decoding time of large models.

[0075] Step 202: Based on multiple retrieval databases, reason about the second input text using a large language model to obtain the reasoning result of the large language model.

[0076] After constructing multiple retrieval databases, speculative decoding techniques can be used to determine the second input text s = {x1, x2, ..., x...}. t Multiple draft texts are identified from the retrieval database. These draft texts are possible inference results obtained by the large language model from the second input text, i.e., the draft texts are candidate texts for the inference results.

[0077] Optionally, a target retrieval database can be determined from multiple retrieval databases based on the second input text. Then, the last N characters of the second input text can be used as a prefix, where N is a positive integer. Based on this prefix, a subtree is determined from the prefix tree of the target retrieval database. This subtree is also a prefix tree, and it includes multiple draft texts, with each branch representing a draft text.

[0078] Optionally, after obtaining the subtree, a depth-first search algorithm can be used to analyze the subtree to obtain the pseudo-sequence and the attention mask of the tree structure. Alternatively, a prefix search algorithm or other retrieval algorithms can be used to analyze the subtree; the specific method is not limited here. The pseudo-sequence comprises characters from multiple draft texts and can be represented as follows: Attention masks are used to obtain draft text from pseudo-sequences.

[0079] like Figure 3As shown, for example, the second input text is "a cat sits", where s = {a, cat, sits}. Taking "sits" as a prefix, a subtree can be determined from a prefix tree in the retrieved database. From this subtree, it can be determined that the subsequent possible draft text could be "on my knee", "on atable", or "on a chair". Based on the determined subtree, the resulting pseudo-sequence could be {on, my, knee, a, table, chair}. Furthermore... Figure 3 The tree-like attention mask shown has black parts representing characters displayed in the pseudo-sequence and white parts representing characters masked in the pseudo-sequence. Therefore, multiple draft texts can be represented based on the pseudo-sequence and the attention mask.

[0080] Optionally, after obtaining the pseudo-sequence and attention mask, the second input text, pseudo-sequence, and attention mask can be analyzed using a large language model based on fuzzy verification conditions to obtain the inference result. The fuzzy verification conditions are used to determine whether to accept characters from the pseudo-sequence based on the conditional probabilities of characters in multiple character positions within the vocabulary.

[0081] Specifically, a large language model can be used to analyze the second input text, the pseudo-sequence, and the attention mask to obtain the conditional probabilities of all characters in the vocabulary at multiple character positions. These character positions are the positions corresponding to the characters obtained from the second input text and the pseudo-sequence. When the pseudo-sequence contains m characters, there are m character positions. The conditional probability of a character in the vocabulary at the i-th character position is the probability that the character in the vocabulary will be output at the i-th character position based on the current context. The formula for calculating the conditional probability of a character in the vocabulary at the i-th character position satisfies... Where i is a positive integer from 1 to m, x represents any character in the vocabulary, and the current context includes... Multiple characters. Subsequently, the pseudo-sequence can be verified based on the fuzzy verification conditions and the conditional probabilities of all characters in the vocabulary at multiple character positions, thus obtaining the inference result.

[0082] In this embodiment, pseudo-sequences are verified based on fuzzy verification conditions and conditional probabilities, which enables the verification of multiple draft texts at once. Through parallel verification processing, the generation time of large language models can be shortened.

[0083] Typically, strict verification conditions are applied to draft text obtained through speculative decoding. This means that only if the character with the highest conditional probability at the current position matches the corresponding character in the draft text can the text at that position in the draft text be accepted. However, since recommendation models do not require perfectly matching text as input, this embodiment can verify the characters at the corresponding positions in the pseudo-sequence based on the k characters with the highest conditional probabilities from the current position. This allows the large language model to accept more characters, improving the character acceptance rate. Simultaneously, controlling the probability conditions effectively prevents divergent generation, thus controlling the length of the generated text and avoiding excessively long text. Here, k can typically be a number between 3 and 5, with no specific limitation on its value. Furthermore, when the predecessor node of the current node in the draft text is not accepted, the current node does not need to be verified, reducing verification time and accelerating the inference of the large language model.

[0084] Optionally, when the first character is any one of the k second characters, the first character can be used as a candidate character for the inference result. Here, the first character is the character corresponding to the i-th position in the pseudo-sequence, and the k second characters are the characters corresponding to the highest conditional probabilities at the i-th position in the vocabulary. After verifying the pseudo-sequence to obtain multiple candidate characters, the longest branch formed by the candidate characters can be used as the inference result and output according to the order of the candidate characters in the subtree. Alternatively, other branches can be selected as the inference result, i.e., the longest branch can be omitted. The specific choice is not limited here.

[0085] Specifically, the conditional probability of each character in the vocabulary at the i-th character position can be calculated. Then, the multiple conditional probabilities are sorted from high to low, and the characters corresponding to the first k conditional probabilities can be used as k second characters. Optionally, the k characters corresponding to the first k conditional probabilities can be further filtered by judging whether the conditional probability of the k characters is greater than a preset probability. If the conditional probability is less than the preset probability, the character cannot be used as a second character. The number of multiple second characters obtained is less than or equal to k.

[0086] Optionally, when the first character is not any of the k second characters, the character with the highest conditional probability in the vocabulary can be used as the character at the current position.

[0087] For example, consider the pseudo-sequence {on, my, knee, a, table, chair}, which contains 6 characters. First, based on the second input text s = {a, cat, sits}, we can obtain the conditional probability of each character in the vocabulary at the first character position. Then, we can verify the character 'on' at the first position in the pseudo-sequence based on this conditional probability. We can determine if 'on' is any of the k second characters. If so, we accept the character 'on' at the first position in the pseudo-sequence. Similarly, if 'my,' knee, and 'a' in the pseudo-sequence are all accepted, and 'table' and 'chair' are not among the k highest-probability characters corresponding to their respective positions, then 'table' and 'chair' are not accepted. In this case, {on, my, knee} can be used as the inference result. The large language model generates 3 tokens at once, increasing the token acceptance rate and accelerating the inference process.

[0088] The text knowledge obtained based on the model reasoning method proposed in this application can be converted into dense vectors by an encoder for use in training downstream recommendation models or other models.

[0089] Figure 4 This is a schematic diagram illustrating the execution process of the model reasoning method proposed in the embodiments of this application. Figure 4 First, multiple retrieval pools can be constructed based on the knowledge data corresponding to users and items. Based on the input text, multiple draft texts are obtained by retrieving data from these pools. For example, if the input text is "the user," the draft text would be "lovesthriller films." Then, fuzzy validation can be used. A large language model is employed to validate the draft texts. At the first character position, the conditional probabilities of all characters in the vocabulary at the first position are sorted from high to low. The characters corresponding to the two highest conditional probabilities are selected to validate the characters in the draft text. Figure 4 As we know, the conditional probabilities of "like" and "loves" in the vocabulary are among the first two conditional probabilities of all characters. The "loves" in the draft text belongs to either "like" or "loves," therefore this character can be accepted. Finally, the output can be the three characters: "loves thriller films."

[0090] For example, based on the Movielens-10M and Amazon-Books datasets, we compared and analyzed the average accepted token, speed, retrieval time, total time, and speed-up using Vanilla, single pool, random pools, customized pools (CP), relaxed verify (RV), and CP+RV (the model inference method proposed in this scheme). The comparison results are shown in Table 1.

[0091] Table 1

[0092]

[0093] As shown in Table 1, our proposed solution achieves speedups of x4.52 and x4.75 on the ML-10M and Amazon-books datasets, respectively, demonstrating significant acceleration. The retrieval database constructed in this solution significantly reduces retrieval time, and the fuzzy validation proposed in this solution can significantly increase the number of accepted character tokens. Furthermore, since fuzzy validation generates text different from the original model, the effects of the knowledge generated under both precise and fuzzy validation strategies on downstream tasks are compared. The comparison results are shown in Table 2. As shown in Table 2, fuzzy validation does not affect the performance of downstream tasks.

[0094] Table 2

[0095]

[0096] The method flow provided in this application has been described above. The apparatus provided in this application will now be described based on the aforementioned method flow.

[0097] See Figure 5 The structural schematic diagram of a model inference device provided in this application is as follows.

[0098] The construction module 501 is used to construct multiple retrieval databases based on multiple text data. The text data is obtained by reasoning about the first input text through a large language model. The similarity between the text data in the retrieval database is higher than a first preset value.

[0099] The reasoning module 502 is used to reason about the second input text based on multiple retrieval databases and a large language model to obtain the reasoning result of the large language model.

[0100] In one possible implementation, the aforementioned construction module 501 is specifically used to: divide multiple text data into multiple sets of data based on the similarity of the multiple text data, wherein the similarity between the text data in each set is higher than a second preset value; and construct multiple retrieval databases based on the multiple sets of data, wherein each set of data corresponds to one retrieval database.

[0101] In one possible implementation, the aforementioned inference module 502 is specifically used for: analyzing multiple retrieval databases based on the second input text to obtain a pseudo-sequence and an attention mask, wherein the pseudo-sequence includes characters from multiple draft texts, and the draft texts are candidate texts for the inference result; and analyzing the second input text, the pseudo-sequence, and the attention mask using a large language model based on fuzzy verification conditions to obtain the inference result, wherein the fuzzy verification conditions are used to determine whether to accept characters in the pseudo-sequence based on the conditional probabilities of characters in multiple character positions in the vocabulary, and the character positions are the positions corresponding to the characters obtained based on the second input text and the pseudo-sequence.

[0102] In one possible implementation, the aforementioned reasoning module 502 is specifically used to: determine a subtree from a target retrieval database based on the second input text, the subtree representing multiple draft texts, the target retrieval database being determined from multiple retrieval databases based on the second input text; and analyze the subtree using a depth-first search algorithm to obtain pseudo-sequences and attention masks.

[0103] In one possible implementation, the aforementioned inference module 502 is specifically used to: analyze the second input text, pseudo-sequence, and attention mask using a large language model to obtain the conditional probabilities of characters in the vocabulary at multiple character positions; and verify the pseudo-sequence based on the fuzzy verification conditions and conditional probabilities to obtain the inference result.

[0104] In one possible implementation, the aforementioned reasoning module 502 is specifically used to: when the first character is any one of the k second characters, take the first character as a candidate character for the reasoning result, where the first character is the character corresponding to the i-th character position in the pseudo-sequence, the k second characters are the characters corresponding to the highest conditional probabilities at the i-th character position in the vocabulary, i is a positive integer from 1 to m, m is the number of characters in the pseudo-sequence, and k is a positive integer; and determine the reasoning result based on the candidate character.

[0105] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method provided in this application.

[0106] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the method provided in this application.

[0107] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, the specific working process of the systems, devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0110] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0114] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A model reasoning method, characterized in that, include: Multiple retrieval databases are constructed based on multiple text data, wherein the text data is obtained by reasoning from the first input text through a large language model, and the similarity between the text data in the retrieval databases is higher than a first preset value; Based on the multiple retrieval databases, the second input text is inferred using the large language model to obtain the inference result of the large language model.

2. The method according to claim 1, characterized in that, The construction of multiple retrieval databases based on multiple text data includes: Based on the similarity of the multiple text data, the multiple text data are divided into multiple groups of data, and the similarity between the text data in each group is higher than a second preset value. Based on the multiple sets of data, the multiple retrieval databases are constructed, with each set of data corresponding to one retrieval database.

3. The method according to claim 1 or 2, characterized in that, The step of reasoning about the second input text using the large language model based on multiple retrieval databases to obtain the reasoning result of the large language model includes: Based on the second input text, the multiple retrieval databases are analyzed to obtain pseudo-sequences and attention masks. The pseudo-sequences include characters from multiple draft texts, which are candidate texts for the inference results. Based on the fuzzy verification conditions, the second input text, the pseudo-sequence, and the attention mask are analyzed by the large language model to obtain the inference result. The fuzzy verification conditions are used to determine whether to accept the characters in the pseudo-sequence based on the conditional probabilities of the characters in the vocabulary at multiple character positions. The character positions are the positions corresponding to the characters obtained from the second input text and the pseudo-sequence.

4. The method according to claim 3, characterized in that, The step of analyzing the multiple retrieval databases based on the second input text to obtain pseudo-sequences and attention masks includes: Based on the second input text, a subtree is determined from the target retrieval database, the subtree being used to represent the plurality of draft texts, and the target retrieval database being determined from the plurality of retrieval databases based on the second input text; The subtree is analyzed using the depth-first search algorithm to obtain the pseudo-sequence and the attention mask.

5. The method according to claim 3 or 4, characterized in that, The step of analyzing the second input text, the pseudo-sequence, and the attention mask using the large language model based on the fuzzy verification conditions to obtain the inference result includes: The second input text, the pseudo-sequence, and the attention mask are analyzed using the large language model to obtain the conditional probabilities of characters in the vocabulary at the multiple character positions. The pseudo-sequence is verified based on the fuzzy verification conditions and the conditional probabilities to obtain the inference result.

6. The method according to claim 5, characterized in that, The step of verifying the pseudo-sequence based on the fuzzy verification conditions and the conditional probability to obtain the inference result includes: When the first character is any one of the k second characters, the first character is taken as a candidate character of the reasoning result. The first character is the character corresponding to the i-th character position in the pseudo-sequence. The k second characters are the characters corresponding to the highest conditional probabilities at the i-th character position in the vocabulary. i is a positive integer from 1 to m. m is the number of characters in the pseudo-sequence. k is a positive integer. Based on the candidate characters, the reasoning result is determined.

7. A model reasoning device, characterized in that, include: The construction module is used to construct multiple retrieval databases based on multiple text data, wherein the text data is obtained by reasoning on the first input text through a large language model, and the similarity between the text data in the retrieval database is higher than a first preset value; The reasoning module is used to reason about the second input text based on the multiple retrieval databases and the large language model to obtain the reasoning result of the large language model.

8. The apparatus according to claim 7, characterized in that, The building module is specifically used for: Based on the similarity of the multiple text data, the multiple text data are divided into multiple groups of data, and the similarity between the text data in each group is higher than a second preset value. Based on the multiple sets of data, the multiple retrieval databases are constructed, with each set of data corresponding to one retrieval database.

9. The apparatus according to claim 7 or 8, characterized in that, The reasoning module is specifically used for: Based on the second input text, the multiple retrieval databases are analyzed to obtain pseudo-sequences and attention masks. The pseudo-sequences include characters from multiple draft texts, which are candidate texts for the inference results. Based on the fuzzy verification conditions, the second input text, the pseudo-sequence, and the attention mask are analyzed by the large language model to obtain the inference result. The fuzzy verification conditions are used to determine whether to accept the characters in the pseudo-sequence based on the conditional probabilities of the characters in the vocabulary at multiple character positions. The character positions are the positions corresponding to the characters obtained from the second input text and the pseudo-sequence.

10. The apparatus according to claim 9, characterized in that, The reasoning module is specifically used for: Based on the second input text, a subtree is determined from the target retrieval database, the subtree being used to represent the plurality of draft texts, and the target retrieval database being determined from the plurality of retrieval databases based on the second input text; The subtree is analyzed using the depth-first search algorithm to obtain the pseudo-sequence and the attention mask.

11. The apparatus according to claim 9 or 10, characterized in that, The reasoning module is specifically used for: The second input text, the pseudo-sequence, and the attention mask are analyzed using the large language model to obtain the conditional probabilities of characters in the vocabulary at the multiple character positions. The pseudo-sequence is verified based on the fuzzy verification conditions and the conditional probabilities to obtain the inference result.

12. The apparatus according to claim 11, characterized in that, The reasoning module is specifically used for: When the first character is any one of the k second characters, the first character is taken as a candidate character of the reasoning result. The first character is the character corresponding to the i-th character position in the pseudo-sequence. The k second characters are the characters corresponding to the highest conditional probabilities at the i-th character position in the vocabulary. i is a positive integer from 1 to m. m is the number of characters in the pseudo-sequence. k is a positive integer. Based on the candidate characters, the reasoning result is determined.

13. A model reasoning device, characterized in that, include: A processor and a memory, wherein the processor is coupled to the memory; The memory is used to store programs; The processor is configured to execute a program in the memory, causing the method as described in any one of claims 1 to 6 to be performed.

14. A computer-readable storage medium comprising instructions that, when executed on a computer, cause the computer to perform the method as claimed in any one of claims 1 to 6.

15. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1 to 6.

Images