Language model inference method and device based on hierarchical state memory bank and combined initialization, terminal, medium and product

Abstract

The application provides a language model reasoning method and device based on hierarchical state memory bank and combined initialization, a terminal, a medium and a product. The method comprises the following steps: constructing a hierarchical state memory bank; obtaining user query information, and performing hierarchical retrieval in the hierarchical state memory bank based on the user query information to obtain a best state set; performing combined initialization on the best state set to generate an initial state; and enabling a basic language model to start reasoning on the user query information in the initial state to generate a corresponding reasoning result. Through hierarchical retrieval, the application can provide matched context for queries of different granularities, solve the cold start problem of the existing linear attention language model in the form of RNN, and make the reasoning result of the model more accurate.

Description

Technical Field

[0001] This application relates to the field of language model technology, and in particular to a language model reasoning method, device, terminal, medium and product based on a hierarchical state memory bank and combinatorial initialization. Background Technology

[0002] Linear attention language models in the form of RNNs are extremely efficient at processing sequences, but their standard implementation suffers from the cold start problem. That is, at the beginning of each task, the model's loop state starts from a zero vector or a random vector, lacking any background knowledge related to the current query. Summary of the Invention

[0003] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a language model reasoning method, device, terminal, medium and product based on hierarchical state memory and combinatorial initialization, to solve the cold start problem of existing RNN-based linear attention language models.

[0004] To achieve the above and other related objectives, a first aspect of this application provides a language model reasoning method based on a hierarchical state memory and combinatorial initialization, comprising: constructing a hierarchical state memory; obtaining user query information, and performing a hierarchical retrieval in the hierarchical state memory based on the user query information to obtain an optimal state set; performing combinatorial initialization on the optimal state set to generate an initial state; and instructing a basic language model to reason about the user query information starting from the initial state to generate a corresponding reasoning result.

[0005] In some embodiments of the first aspect of this application, the construction of the hierarchical state memory includes: performing hierarchical clustering on a target document corpus transformed by a latent vector model to generate a hierarchical vector library; and using a basic language model to perform state transformation on the hierarchical vector library to obtain the hierarchical state memory.

[0006] In some embodiments of the first aspect of this application, the hierarchical state memory adopts a tree structure; each node of the hierarchical state memory carries corresponding category information and a cyclic state tensor.

[0007] In some embodiments of the first aspect of this application, based on the user query information, a hierarchical retrieval is performed in the hierarchical state memory to obtain an optimal state set, including: transforming the latent vector model of the user query information to obtain a corresponding semantic vector; based on the semantic vector, starting from the root node of the hierarchical state memory, comparing and selecting layer by layer downwards to determine an optimal path from the root node to the most relevant node; and outputting the cyclic state tensors corresponding to all nodes on the optimal path as the optimal state set.

[0008] In some embodiments of the first aspect of this application, the most relevant node is determined according to a predetermined traversal level; the traversal level is related to the category of the user query information.

[0009] In some embodiments of the first aspect of this application, the optimal state set is subjected to combinatorial initialization to generate an initial state, including: weighted fusion of each loop state tensor in the optimal state set based on the Fuse function to obtain the initial state.

[0010] To achieve the above and other related objectives, a second aspect of this application provides a language model inference apparatus based on a hierarchical state memory and combinatorial initialization, comprising: a construction module for constructing a hierarchical state memory; a hierarchical retrieval module for acquiring user query information and performing a hierarchical retrieval in the hierarchical state memory based on the user query information to obtain an optimal set of states; a combinatorial initialization module for performing combinatorial initialization on the optimal set of states to generate an initial state; and an inference module for instructing a basic language model to infer the user query information starting from the initial state and generate a corresponding inference result.

[0011] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the language model reasoning method based on a hierarchical state memory and combinatorial initialization.

[0012] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code, which, when executed on a computer, enables the computer to implement the language model reasoning method based on a hierarchical state memory and combinatorial initialization.

[0013] To achieve the above and other related objectives, a fifth aspect of this application provides an electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the language model reasoning method based on a hierarchical state memory and combinatorial initialization.

[0014] As described above, the language model reasoning method, apparatus, terminal, medium, and product based on hierarchical state memory and combinatorial initialization of this application have the following beneficial effects: Through hierarchical retrieval, this application can provide matching context for queries of different granularities, solve the cold start problem of existing RNN-type linear attention language models, and make the inference results of the model more accurate. Attached Figure Description

[0015] Figure 1 The diagram shown is a flowchart illustrating a language model reasoning method based on a hierarchical state memory bank and combinatorial initialization in one embodiment of this application.

[0016] Figure 2 The diagram shown is a schematic representation of a tree structure in one embodiment of this application.

[0017] Figure 3 The diagram shown is a schematic block diagram of a language model reasoning device based on a hierarchical state memory bank and combinatorial initialization in one embodiment of this application.

[0018] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0019] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0020] In the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and that the terms "first" and "second" do not necessarily imply that they are different.

[0021] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0022] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0023] To facilitate understanding of the embodiments of this application, firstly, in conjunction with Figure 1 Detailed explanation. Figure 1 This document illustrates a flowchart of a language model inference method based on a hierarchical state memory and combinatorial initialization, as described in an embodiment of the present invention. The language model inference method based on a hierarchical state memory and combinatorial initialization in this embodiment mainly includes the following steps:

[0024] Step S101: Construct a hierarchical state memory library.

[0025] In one embodiment, the construction of the hierarchical state memory includes: performing hierarchical clustering on the target document corpus transformed by the latent vector model to generate a hierarchical vector library; and using a basic language model to perform state transformation on the hierarchical vector library to obtain the hierarchical state memory.

[0026] Specifically, a large-scale document corpus is acquired as the target corpus. A large-scale document corpus refers to a collection containing massive amounts of structured / unstructured text data, used to train Natural Language Processing (NLP) models, support text mining, information retrieval, semantic analysis, and other tasks. Its core characteristics are large scale, broad coverage, and clearly defined labeled / unlabeled attributes. Large-scale document corpora cover various types of subject knowledge, making subsequent hierarchical retrieval results more reliable. Next, a latent vector model is used to convert the large-scale document corpus into corresponding embedding vectors. It should be understood that an embedding vector is a low-dimensional, dense real-number vector used to map discrete, high-dimensional objects (such as words, sentences, images, user IDs, etc.) to a continuous, low-dimensional vector space. The core purpose is to preserve the semantic or feature relationships of the original objects. It should also be understood that a latent vector model is a machine learning model that captures the potential relationships between entities by mapping entities (such as users, items, words, etc.) to low-dimensional, dense vectors (i.e., latent vectors / embedded vectors).

[0027] Furthermore, hierarchical clustering is performed on the aforementioned embedding vectors to obtain a hierarchical vector library, along with cluster centers. At this point, the hierarchical vector library is a knowledge base employing a tree structure. The hierarchical vector library comprises multiple levels, and each parent node in the library has at least one child node. The hierarchical vector library has only one root node, which represents global knowledge. Each node in the hierarchical vector library carries corresponding category information and an embedding cluster center. The category information and embedding cluster center carried by each node are related to the level at which the node belongs. The category of a parent node is the higher-order category of its child nodes. For example, for a four-level tree structure, such as... Figure 2 As shown, the root node corresponds to the "global" category information. The next level of nodes can correspond to categories such as "science" or "history," and the next level can correspond to sub-disciplines such as "physics" or "chemistry." The category information for each node in the final level is "specific knowledge," corresponding to a specific document or knowledge point. It should be understood that hierarchical clustering can use the hierarchical k-means algorithm or other specific algorithms, depending on the requirements; no limitation is made here. The cluster center is the core representative point of all sample data within each cluster in the clustering algorithm. Its role is to quantify the overall characteristics of the cluster; it is the center point or mean point of the samples within the cluster in the feature space.

[0028] Furthermore, the embedding vector carried by each node in the hierarchical vector library is input into the basic language model to obtain the corresponding cyclic state tensor. Simultaneously, continuing the tree structure of the hierarchical vector library, a hierarchical state memory is constructed. Each node in the hierarchical state memory carries corresponding category information and a cyclic state tensor. Any node in the hierarchical state memory has the same category as its corresponding node in the hierarchical vector library. It should be understood that the basic language model is a linear attention language model in the form of an RNN. The state tensor is the tensor used by the RNN-type linear attention language model to store memory when processing sequential data; it records the background information obtained by the model after processing previous sequence elements.

[0029] In one embodiment, the hierarchical state memory can be built offline.

[0030] Step S102: Obtain user query information, and based on the user query information, perform a hierarchical retrieval in the hierarchical state memory to obtain the optimal state set.

[0031] In one embodiment, based on the user query information, a hierarchical retrieval is performed in the hierarchical state memory to obtain the optimal state set, including: transforming the user query information through a latent vector model to obtain the corresponding semantic vector; based on the semantic vector, starting from the root node of the hierarchical state memory, comparing and selecting layer by layer downwards to determine the optimal path from the root node to the most relevant node; and outputting the cyclic state tensors corresponding to all nodes on the optimal path as the optimal state set.

[0032] In one embodiment, the most relevant node is determined according to a predetermined traversal level; the traversal level is related to the category of the user query information.

[0033] Specifically, after obtaining the user query information, the query information is first converted into a corresponding semantic vector (embedding vector) using a latent vector model (which has the same function as the latent vector model mentioned above, and will not be elaborated here). The similarity between the semantic vector and the obtained cluster centers is calculated, and the category of the cluster center with the highest similarity is determined as the category of the user query information. Then, the traversal level corresponding to the category of the user query information is queried from the pre-constructed category hierarchy mapping relationship.

[0034] Furthermore, starting from the root node of the hierarchical state memory, comparisons and selections are performed layer by layer downwards. For any layer, the similarity between the semantic vector and all nodes in that layer is calculated, and the node with the highest similarity in that layer is selected as a candidate node. When comparing layers corresponding to all levels, the node with the highest similarity in that layer is the most relevant node. The root node, each candidate node, and the most relevant node constitute the optimal path.

[0035] Finally, the cyclic state tensors corresponding to each node in the optimal path are output as the optimal state set.

[0036] Step S103: Perform combinatorial initialization on the set of optimal states to generate initial states.

[0037] In one embodiment, the optimal state set is subjected to combinatorial initialization to generate an initial state, including: weighted fusion of each loop state tensor in the optimal state set based on the Fuse function to obtain the initial state.

[0038] Specifically, the set of optimal states is represented as , where k is a positive integer. Then, the optimal state set is initialized combinatorially according to the following formula 1:

[0039] (1)

[0040] in, It can be preset or learned dynamically through the model to emphasize the importance of different levels.

[0041] Step S104: Instruct the basic language model to start reasoning on the user query information in the initial state and generate the corresponding reasoning results.

[0042] First, let's explain the characteristics of basic language models. Basic language models employ a linear attention mechanism and perform inference in the form of an RNN. The core of this type of language model is transforming attention computation into linear recursive state updates. Typical examples of this type of language model include Linear Transformer, RWKV, TransNormerLLM, and Delta Net. During inference, only a small number of state variables need to be maintained, making it suitable for processing extremely long texts. This type of language model can be viewed as a type of RNN model. The core of an RNN model is the hidden state (h), which is responsible for passing information between time steps. Before processing the input of the first time step, the RNN model has no historical information, therefore an initial value needs to be set for the hidden state (h). This is the initial state of the RNN model.

[0043] Based on the aforementioned model characteristics, step S104 is implemented as follows: the initial state generated in step S103 is set as the initial value of the basic language model. This ensures that the initial state obtained by the basic language model is no longer an isolated knowledge point, but rather a knowledge structure containing subordinate relationships and systematic organization, making the model's reasoning more accurate. Then, the user query information is input into the basic language model, enabling the basic language model to reason about the user query information using the highly information-rich initial state obtained through combination, and to obtain the final reasoning result.

[0044] Next, the process of constructing the category hierarchy mapping relationship is explained below:

[0045] First, obtain query information datasets for multiple categories. The categories of the datasets correspond to the categories of the hierarchical state memory. Each category of the query information dataset includes multiple query messages within that category. For each category of the query information dataset, assume it contains N query messages. For the i-th query message in that category of the query information dataset, Where i is a positive integer. The i-th query result is converted into a corresponding semantic vector using a latent vector model. Meanwhile, assuming the hierarchical state memory has an M-layer structure, for semantic vectors... Perform level 1, level 2, ..., level M traversals respectively. For level j ( Traverse the hierarchical state memory (where j is a positive integer), starting from the root node, and calculate the semantic vector. The similarity with each node in each layer is used to output the cyclic state tensor corresponding to the node with the highest similarity in each layer from layer 1 to layer j as the optimal state set for layer j traversal. Then, the optimal state set for layer j traversal is initialized using the combinatorial method mentioned in step S103 to obtain the initial state for layer j traversal. The initial states of layer 1 traversal, layer 2 traversal, ..., layer M traversal are set as the initial values ​​of the basic language model, so that the basic language model can infer the i-th query information and obtain the inference results corresponding to layer 1 traversal, layer 2 traversal, ..., layer M respectively. Among the M inference results, the inference result that best matches the i-th query information is identified, and the corresponding traversal layer number is recorded as the traversal layer number of the i-th query information. It should be understood that the best matching inference result can be determined according to the requirements, and is not limited here.

[0046] Calculate the average number of traversal levels for all query messages in the query information dataset for this category, and use this average as the traversal level for this category.

[0047] Other categories of query data datasets are processed in the same way. All categories and their corresponding traversal levels are output as a category hierarchy mapping.

[0048] It should be understood that the similarity mentioned in all the above embodiments can be calculated using existing similarity methods, which will not be elaborated here.

[0049] To better illustrate the method of the present invention, a specific embodiment is now provided.

[0050] The system retrieves the user query "What is the core principle of Shor's algorithm?", and then performs a path traversal in the hierarchical state memory. The determined optimal path is [root node]. science computer Science Quantum computing Shor's algorithm]. Then, all the cyclic state tensors along the path are output as the optimal state set. .

[0051] Perform combinatorial initialization for the optimal state. The weights here are illustrative, indicating that for this specific problem, the focus is more on the details of Shor's algorithm itself, but a certain background knowledge of quantum computing is required to support the explanation. The final result... He is an expert who understands both the macroscopic concepts of quantum computing and the specific principles of Shor's algorithm.

[0052] Basic language model Let's begin reasoning with the initial state. Because... It already contains systematic knowledge, and the model can generate very high-quality answers, such as: "The core principle of Shor's algorithm is to use the parallelism of quantum computing to efficiently find the period of a function through quantum Fourier transform, and this period can be used to derive the prime factors of large numbers..." This answer naturally explains the specific technology (quantum Fourier transform) within its macroscopic domain (quantum computing parallelism).

[0053] The advantages of this invention are:

[0054] (1) Adaptive context granularity: Through hierarchical retrieval, this invention can provide matching context for queries of different granularities. Broad queries may only use the upper-level state of the tree, while specific queries will use the deeper state. The combined initialization mechanism ensures that the depth and breadth of the context are just right.

[0055] (2) Systematic knowledge injection: The initial state obtained by the model is no longer an isolated knowledge point, but a knowledge structure containing subordinate relationships and systematic organization. This makes the model's answer logic more rigorous and organized.

[0056] (3) High retrieval efficiency: In a memory bank with massive states, hierarchical tree retrieval reduces the search complexity from linear level. Reduced to logarithmic level This makes the state retrieval process extremely fast.

[0057] (4) Zero generation overhead and hot start: This invention maintains zero generation overhead in the knowledge injection process, achieves true hot start, and retains the original inference efficiency of the model to the maximum extent. No hardware upgrade is required. Better model performance can be obtained with the same computing resources, thus improving the hardware energy efficiency ratio.

[0058] Figure 3 This is a schematic block diagram of a language model inference device based on a hierarchical state memory and combinatorial initialization, provided in an embodiment of this application. Figure 3 As shown, the language model inference device 300 based on a hierarchical state memory bank and combinatorial initialization includes:

[0059] Module 301 is used to build a hierarchical state memory library;

[0060] The hierarchical retrieval module 302 is used to obtain user query information and, based on the user query information, perform hierarchical retrieval in the hierarchical state memory to obtain the optimal state set.

[0061] The combined initialization module 303 is used to perform combined initialization on the set of optimal states to generate an initial state;

[0062] The reasoning module 304 is used to enable the basic language model to start reasoning on the user query information in the initial state and generate the corresponding reasoning results.

[0063] It should be understood that the specific process of each module performing the above-mentioned steps has been described in detail in the above method embodiments, and will not be repeated here for the sake of brevity.

[0064] It should also be understood that the module division in the embodiments of this application is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0065] In one embodiment, the construction of the hierarchical state memory includes: performing hierarchical clustering on the target document corpus transformed by the latent vector model to generate a hierarchical vector library; and using a basic language model to perform state transformation on the hierarchical vector library to obtain the hierarchical state memory.

[0066] In one embodiment, the hierarchical state memory adopts a tree structure; each node of the hierarchical state memory carries corresponding category information and a cyclic state tensor.

[0067] In one embodiment, based on the user query information, a hierarchical retrieval is performed in the hierarchical state memory to obtain the optimal state set, including: transforming the user query information through a latent vector model to obtain the corresponding semantic vector; based on the semantic vector, starting from the root node of the hierarchical state memory, comparing and selecting layer by layer downwards to determine the optimal path from the root node to the most relevant node; and outputting the cyclic state tensors corresponding to all nodes on the optimal path as the optimal state set.

[0068] In one embodiment, the most relevant node is determined according to a predetermined traversal level; the traversal level is related to the category of the user query information.

[0069] In one embodiment, the optimal state set is subjected to combinatorial initialization to generate an initial state, including: weighted fusion of each loop state tensor in the optimal state set based on the Fuse function to obtain the initial state.

[0070] Figure 4 This is a schematic block diagram of the electronic terminal provided in an embodiment of this application. Figure 4 As shown, the electronic terminal includes at least one processor 401, a memory 402, at least one network interface 403, and a user interface 405. The various components in the device are coupled together via a bus system 404. It is understood that the bus system 404 is used to implement communication between these components. In addition to a data bus, the bus system 404 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 4 The general will label all buses as bus systems.

[0071] The user interface 405 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0072] It is understood that memory 402 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0073] In this embodiment of the invention, the memory 402 is used to store various types of data to support the operation of the electronic terminal 400. Examples of this data include any executable program for operation on the electronic terminal 400, such as the operating system 4021 and application programs 4022. The operating system 4021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 4022 may contain various applications, such as a media player, browser, etc., for implementing various application services. The language model reasoning method based on hierarchical state memory and combinatorial initialization provided in this embodiment of the invention can be included in the application program 4022.

[0074] The methods disclosed in the above embodiments of the present invention can be applied to processor 401, or implemented by processor 401. Processor 401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 401 or by instructions in the form of software. The processor 401 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 401 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. General-purpose processor 401 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0075] In an exemplary embodiment, the electronic terminal 400 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to execute the aforementioned method.

[0076] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figure 1 The language model reasoning method based on hierarchical state memory and combinatorial initialization in the illustrated embodiment.

[0077] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when executed on a computer, causes the computer to perform... Figure 1 The language model reasoning method based on hierarchical state memory and combinatorial initialization in the illustrated embodiment.

[0078] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0079] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0080] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0082] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0083] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0084] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0085] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0086] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0087] In summary, this application provides a language model inference method, apparatus, terminal, medium, and product based on a hierarchical state memory and combinatorial initialization. The method includes: constructing a hierarchical state memory; acquiring user query information and, based on the user query information, performing a hierarchical retrieval in the hierarchical state memory to obtain an optimal state set; performing combinatorial initialization on the optimal state set to generate an initial state; and instructing a basic language model to infer the user query information starting from the initial state, generating corresponding inference results. This application, through hierarchical retrieval, can provide matching context for queries of different granularities, solving the cold-start problem of existing RNN-based linear attention language models, making the model's inference results more accurate. Therefore, this application effectively overcomes various shortcomings of the prior art and has high industrial applicability.

[0088] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A language model reasoning method based on a hierarchical state memory bank and combinatorial initialization, characterized in that, include: A hierarchical state memory is constructed. The construction method of the hierarchical state memory includes: performing hierarchical clustering on the target document corpus transformed by the latent vector model to generate a hierarchical vector library; using a basic language model to perform state transformation on the hierarchical vector library to obtain the hierarchical state memory; the hierarchical state memory adopts a tree structure; each node of the hierarchical state memory carries corresponding category information and a cyclic state tensor. The process involves acquiring user query information and performing a hierarchical retrieval in the hierarchical state memory based on that information to obtain an optimal set of states. This process includes: transforming the user query information using a latent vector model to obtain corresponding semantic vectors; determining an optimal path from the root node to the most relevant node by comparing and selecting nodes layer by layer based on the semantic vectors; and outputting the cyclic state tensors corresponding to all nodes on the optimal path as the optimal set of states. The cyclic state tensors are tensors used by a linear attention language model in RNN form to store memory when processing sequence data. The optimal state set is initialized using a combinatorial method to generate the initial state; The basic language model is instructed to begin reasoning about the user query information from the initial state and generate the corresponding reasoning results; wherein the basic language model is a linear attention language model in the form of an RNN.

2. The language model reasoning method based on hierarchical state memory and combinatorial initialization as described in claim 1, characterized in that, The most relevant node is determined based on a pre-determined traversal level; the traversal level is related to the category of the user query information.

3. The language model reasoning method based on hierarchical state memory and combinatorial initialization as described in claim 1, characterized in that, The optimal state set is initialized using a combinatorial method to generate an initial state, including: using the Fuse function to perform weighted fusion of the loop state tensors in the optimal state set to obtain the initial state.

4. A language model reasoning device based on a hierarchical state memory bank and combinatorial initialization, characterized in that, include: A construction module is used to construct a hierarchical state memory library. The construction method of the hierarchical state memory library includes: performing hierarchical clustering on the target document corpus transformed by the latent vector model to generate a hierarchical vector library; using a basic language model to perform state transformation on the hierarchical vector library to obtain the hierarchical state memory library; the hierarchical state memory library adopts a tree structure; each node of the hierarchical state memory library carries corresponding category information and a cyclic state tensor. A hierarchical retrieval module is used to acquire user query information and, based on the user query information, perform a hierarchical retrieval in the hierarchical state memory to obtain an optimal set of states. The process of performing a hierarchical retrieval in the hierarchical state memory to obtain the optimal set of states based on the user query information includes: transforming the user query information using a latent vector model to obtain corresponding semantic vectors; based on the semantic vectors, comparing and selecting layer by layer downwards from the root node of the hierarchical state memory to determine an optimal path from the root node to the most relevant node; and outputting the cyclic state tensors corresponding to all nodes on the optimal path as the optimal set of states. The cyclic state tensors are tensors used by a linear attention language model in RNN form to store memory when processing sequence data. A combined initialization module is used to perform combined initialization on the set of optimal states to generate an initial state; The reasoning module is used to enable the basic language model to start reasoning on the user query information in the initial state and generate the corresponding reasoning results; wherein, the basic language model is a linear attention language model in the form of RNN.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 3.

6. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to implement the method as described in any one of claims 1 to 3.

7. An electronic terminal, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1 to 3.

Images