Tictionary tree-based AI large model Tensor indexing and saving method, system and application

Abstract

The invention discloses an AI large model Tensor indexing and storing method based on a dictionary tree. The method comprises the following steps of 1, dictionary tree construction, wherein root nodes are initialized, nodes are expanded character by character, and key value end points are marked; step 2, preprocessing: formatting the input key value, and removing unnecessary characters; step 3, index construction: a path is created for each unique key value by using a hierarchical structure of the dictionary tree, and each node represents a character in the key value; step 4, step-by-step retrieval: starting from a root node of the dictionary tree, traversing child nodes in sequence according to each character of the key value; and 5, data association: associating or storing a pointer or reference pointing to actual data at a node corresponding to the last character of the key value. The invention further discloses a system and application for implementing the method.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology and relates to a method, system and application for indexing and storing a large AI model Tensor based on a Trie. Background Art

[0002] In recent years, big models have truly become a shining star in the field of artificial intelligence. This is mainly due to two factors: hardware and data. The popularization of the Internet and the process of digitalization have brought about massive data, providing a solid foundation for the learning of big models. In the course of development, some important milestone models have emerged, such as Google's BERT model, OpenAI's GPT series, and Meta's LLaMA model. The advent of these models marks the advent of the era of big models.

[0003] With the rapid improvement of computer performance, especially the emergence of graphics processing units (GPUs) and more specialized AI chips, large models can be trained and inferred more efficiently. In addition, running large model inference directly on high-performance CPUs saves the high cost of purchasing GPGPU or ASIC, and can quickly experience the results of large model inference, which is also an optional solution for experiencing large models. According to recent measured data, the average inference speed of the Q4 Weight quantized version of the LLaMA7 billion parameter model on the AMD EPYC 7543 CPU is 22.39 tokens per second, and the Q4 Weight quantized version of the 13 billion parameter model can also reach 10.32 tokens per second.

[0004] Compared with traditional CNN network models, the number of parameters of Transformer-based artificial intelligence models is very large. For example, the number of parameters of common ChatGPT or LLaMA models is 7 billion, 13 billion, 70 billion, etc., and the trained model weights are hundreds of GB in size. In the process of model reasoning, how to efficiently and quickly find the weight data required by each operator is crucial to improving the efficiency of model reasoning.

[0005] The most typical large model reasoning framework is an open source framework called LLaMA.CPP on Git Hub, which supports reasoning on the global mainstream open source models on the CPU side, such as LLaMA, Falcon, Starcoder, and the domestic large model Baichuan. In fact, the method currently used to index the weights of large models of tens of GB is a single necklace table, which has the following typical disadvantages:

[0006] 1) Inefficient search and deletion operations: In a singly linked list, if you want to delete a node, you need to traverse from the head of the linked list to find the node to be deleted. This will be very time-consuming, especially when the linked list is large;

[0007] 2) Cannot directly access the predecessor node: Since each node of a singly linked list contains only a pointer to the successor node, it is impossible to directly access the predecessor node of a node. Summary of the invention

[0008] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method for indexing and saving AI large model Tensor based on Trie. The method of the present invention can efficiently search and traverse Tensor, efficiently and quickly accelerate the efficiency of model reasoning, and optimize and accelerate the reasoning of artificial intelligence large models on the CPU.

[0009] The AI ​​large model Tensor indexing and storage method based on Trie in the present invention includes the following technical improvements:

[0010] 1. Efficient and fast indexing of Tensor: Compared with the traditional indexing method based on linked lists, this invention proposes an indexing method based on a dictionary tree (Trie), which can efficiently index the weight data required during the operation of large model operators. The improvement data related to its indexing efficiency will be provided later;

[0011] 2. Improvement of the Trie tree: In view of the usage habits of large models for weights, the present invention proposes an improvement method to reasonably cut and prune the Trie tree to further improve the indexing efficiency and reduce the memory space of the Trie tree itself;

[0012] 3.Tensor memory management and optimization: mainly involves the reasonable allocation of memory and the avoidance of memory fragmentation.

[0013] The present invention provides a method for indexing and saving AI large model Tensor based on dictionary tree, and the method comprises the following steps:

[0014] Step 1: Dictionary tree construction: initialize the root node, expand the node character by character, and mark the end point of the key value;

[0015] Step 2: Preprocessing: Format the input key value and remove unnecessary characters;

[0016] Step 3: Build an index: Use the hierarchical structure of the dictionary tree to create a path for each unique key value, where each node represents a character in the key value;

[0017] Step 4: Search level by level: Starting from the root node of the dictionary tree, traverse the child nodes one by one according to each character of the key value;

[0018] Step 5: Data association: At the node corresponding to the last character of the key value, associate or save a pointer or reference to the actual data.

[0019] In step 1, the construction of the dictionary tree includes the following steps:

[0020] Step 1.1, use tensor name as the key value of the dictionary tree, and pre-process the key value to retain legal characters;

[0021] Step 1.2: Build a root node, and based on the root node, expand the child nodes character by character according to the legal characters obtained in step 1.1 to build a complete dictionary tree.

[0022] Step 1.2 also includes the process of pruning the constructed dictionary tree, merging Tensor names with the same prefix and suffix together to reduce the number of nodes in the dictionary tree. The dictionary tree is pruned by setting the number of m_next pointers of the merged nodes to 1 and setting m_is_opt to True;

[0023] In the dictionary tree, the weight data is represented by a Tensor structure, which includes the Tensor name m_name, the Tensor dimension m_dim, the shape of the Tensor in each dimension m_shape, the Tensor gradient change m_stride, the number of Tensor elements m_count, and the Tensor specific data pointer m_data.

[0024] The preprocessing in step 1.1 and the preprocessing in step 2 both refer to removing characters other than legal characters from the output tensor name key value. The legal characters are 26 uppercase and lowercase letters and 10 numeric characters, a total of 62 characters.

[0025] In step 3, each unique Tensor name is used as a key value, and a path in the dictionary tree is created according to the order of the characters in the key value. Each node on the path represents a character in the Tensor name, which is used to retrieve and find Tensor information in the dictionary tree.

[0026] In step 4, during the traversal process, each character is used to match the corresponding child node in the dictionary tree; if the character matches successfully, it moves to the corresponding child node and continues to match the next character until the complete key value is matched or the match cannot continue.

[0027] In step 5, when step 4 is matched successfully and the destination node of the target key value is found, a pointer or reference pointing to the actual data of the Tensor is associated or saved in the destination node, and information or data including the weight Tensor or the activation value Tensor is located;

[0028] The Tensor data is stored in pre-allocated memory space, and the size of the Tensor structure is ensured to be consistent with the memory alignment requirements.

[0029] The present invention also provides applications of the above method in deep learning model reasoning and deployment, model pruning and optimization, and model interpretation.

[0030] The present invention also provides a hardware system for implementing the above method, the hardware system comprising: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the above method is implemented.

[0031] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above method is implemented.

[0032] The beneficial effects of the present invention include:

[0033] 1. When indexing a specific Tensor, the improvement in its time complexity is quite obvious. The time complexity of searching for an element in a single linked list is O(n), where n is the number of elements in the linked list. It is necessary to traverse the entire linked list and compare the values ​​of each node until the target element is found or the end of the linked list is reached; the time complexity of searching for an element in a dictionary (Trie) is O(m), where m is the length of the target string. Specifically for the specific LLaMA model case of Meta in the present invention, n=291+1188, and m=20, it can be seen that the time complexity of the method of the present invention has been significantly reduced.

[0034] 2. Compared with the traditional dictionary (Trie), the present invention has an obvious pruning effect. Specifically, for the LLaMA model case of Meta in the present invention, the common prefix "layer" and the suffix "weight" are merged and pruned respectively, and the compression ratio of the dictionary (Trie) is about 30%.

[0035] 3. Tensors and their data are aligned and stored in pre-allocated memory, which obviously avoids memory fragmentation and also avoids the problem that memory allocation for a single Tensor may result in unsuccessful allocation. Figure 1 The memory space required for each weight tensor is provided in detail. If a large amount of memory is allocated individually, it is very likely to fail due to memory fragmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 Represents the weight Tensor of the LLaMA model and its related information.

[0038] Figure 2 Represents a dictionary (Trie) tree containing 14 key values.

[0039] Figure 3 This example illustrates the actual Node pointing process of a dictionary (Trie) tree containing three key values.

[0040] Figure 4 Shows a relatively impressive pruning effect.

[0041] Figure 5 Describes in detail how the index nodes of the dictionary (Trie) tree and the layout of Tensors and their data in memory. DETAILED DESCRIPTION

[0042] The present invention is further described in detail with reference to the following specific examples and drawings. The process, conditions, experimental methods, etc. for implementing the present invention, except for the contents specifically mentioned below, are all common knowledge and common common sense in the art and are not particularly limited by the present invention.

[0043] The present invention provides a method for indexing and saving AI large model Tensor based on dictionary tree, and the method comprises the following steps:

[0044] Step 1: Dictionary tree construction: initialize the root node, expand the node character by character, and mark the end point of the key value;

[0045] Step 2: Preprocessing: Format the input key value and remove unnecessary characters;

[0046] Step 3: Build an index: Use the hierarchical structure of the dictionary tree to create a path for each unique key value, where each node represents a character in the key value;

[0047] Step 4: Search level by level: Starting from the root node of the dictionary tree, traverse the child nodes one by one according to each character of the key value;

[0048] Step 5: Data association: At the node corresponding to the last character of the key value, associate or save a pointer or reference to the actual data.

[0049] In step 1, the construction of the dictionary tree includes the following steps:

[0050] Step 1.1, use tensor name as the key value of the dictionary tree, and pre-process the key value to retain legal characters;

[0051] Step 1.2: Build a root node, and based on the root node, expand the child nodes character by character according to the legal characters obtained in step 1.1 to build a complete dictionary tree.

[0052] Step 1.2 also includes the process of pruning the constructed dictionary tree, merging Tensor names with the same prefix and suffix together to reduce the number of nodes in the dictionary tree. The dictionary tree is pruned by setting the number of m_next pointers of the merged nodes to 1 and setting m_is_opt to True;

[0053] In the dictionary tree, the weight data is represented by a Tensor structure, which includes the Tensor name m_name, the Tensor dimension m_dim, the shape of the Tensor in each dimension m_shape, the Tensor gradient change m_stride, the number of Tensor elements m_count, and the Tensor specific data pointer m_data.

[0054] The preprocessing in step 1.1 and the preprocessing in step 2 both refer to removing characters other than legal characters from the output tensor name key value. The legal characters are 26 uppercase and lowercase letters and 10 numeric characters, a total of 62 characters.

[0055] In step 3, each unique Tensor name is used as a key value, and a path in the dictionary tree is created according to the order of the characters in the key value. Each node on the path represents a character in the Tensor name, which is used to retrieve and find Tensor information in the dictionary tree.

[0056] In step 4, during the traversal process, each character is used to match the corresponding child node in the dictionary tree; if the character matches successfully, it moves to the corresponding child node and continues to match the next character until the complete key value is matched or the match cannot continue.

[0057] In step 5, when step 4 is matched successfully and the destination node of the target key value is found, a pointer or reference pointing to the actual data of the Tensor is associated or saved in the destination node, and information or data including the weight Tensor or the activation value Tensor is located;

[0058] The Tensor data is stored in pre-allocated memory space, and the size of the Tensor structure is ensured to be consistent with the memory alignment requirements.

[0059] In the present invention, the Tensor structure is used to represent the weight data of the large model: the weight data of the large model and the Tensor in the reasoning process are actually the inputs required by the operator, and can be uniformly represented by the Tensor structure. The difference is that the weight data is static data in the reasoning process, and its input is the activation value, which changes with the change of the input.

[0060] Starting from a specific model, it has X weight Tensors, so X nodes are required to save the corresponding Tensor information. The present invention uses the Tensor name as the index key to create a dictionary tree to index all Tensor information. In addition to the X static weight Tensors in the early stage, the input and output Tensors used to save the activation values ​​in the later stage can also be inserted into the dictionary tree, recorded as Y. In general, the number of Tensors involved in each inference of a Token is X (weight) + Y (activation).

[0061] The specific implementation of Tensor is a C++ structure, which mainly involves several fields including:

[0062] m_name is the name of the tensor, which will also be used as its unique index ID later;

[0063] m_dim, represents the dimension of the current tensor;

[0064] m_shape, represents the shape of the current tensor in each dimension;

[0065] m_stride, represents the gradient change of dimension;

[0066] m_count, indicating the number of elements in the current tensor;

[0067] m_data is a pointer to the specific data of Tensor.

[0068] In a specific implementation, taking Meta's LLaMA model as an example, it has 291 weight Tensors, so 291 nodes are required to save the corresponding Tensor information. The present invention uses the Tensor name as the index key and creates a dictionary tree to index all Tensor information. In addition to the 291 static weight Tensors in the early stage, the input and output tensors used to save the activation values ​​in the later stage can also be inserted into the dictionary tree. In general, the number of Tensors involved in each inference of a Token is 291 (weight) + 1188 (activation).

[0069] In a specific implementation, the construction process of the dictionary tree (Trie) includes the following steps:

[0070] Step 1.1, use the tensor name as the dictionary tree key value and pre-process the key;

[0071] The present invention stipulates that the legal key includes 26 uppercase and lowercase letters and 10 numeric characters, a total of 62 characters. The pre-processing will remove other characters except the above 62 characters. For example, if the name of a tensor is "layers.0.att.wk.weight", it will be processed into "layers0attwkweight" through pre-processing. Figure 2 , which vividly depicts a dictionary (Trie) containing 14 key values. Figure 2 In the middle, the upper part is 14 key values, which can be represented by a dictionary tree.

[0072] Specifically, for example, if a tensor is named "layers.0.att.wk.weight", after preprocessing, all elements that do not belong to the above 62 legal characters (such as period ".") will be removed, so the name of the tensor will be processed into "layers0attwkweight".

[0073] And attached Figure 2 What is displayed is a dictionary tree, which consists of multiple nodes, each node represents a character.

[0074] There are 14 separate keys at the top of the tree, which are processed as tensor names and do not contain illegal characters.

[0075] Each layer of the dictionary tree represents a character of the key name, and the path from the root node to the leaf node represents a complete key name.

[0076] Each branch point in the tree represents a character in the key name, and the existence of a branch indicates that multiple key names share the same prefix.

[0077] Figure 2 The numbers and letters in the may represent characters in the key name or the index of the node.

[0078] A complete dictionary tree (Trie) will have a root node, and then each node will be followed by 62 next nodes, representing the above 62 different characters. Each node is a structure, and the main fields involved are:

[0079] m_is_opt, indicating whether it is an optimized node, which will be used in the process of pruning the dictionary (Trie) tree;

[0080] m_data is a pointer to a Tensor. If the key value on the current path is not a complete Tensor key, the pointer is NULL.

[0081] m_next

[62] is a pointer array pointing to the next character. For details, refer to Figure 3 , which illustrates the process of pointing to the actual node (Node) of the dictionary (Trie) tree containing 3 key values.

[0082] Figure 3 In the example, each node has multiple branches, with arrows indicating the transition from the current character node to the next character node. For example, starting from the root node, if you follow the path labeled with the character "l", this path may point to another node labeled with "a", and so on, until a complete key value is formed.

[0083] Each row represents a level of the dictionary tree, reflecting the construction process of the key-value string from top to bottom. Figure 3 For each key value, start from the root node and proceed along the corresponding character nodes in sequence until the last character of the key value is reached.

[0084] Specifically, taking a key value "layer0attwkweight" as an example, starting from the root node, first locate the node corresponding to the character "l", then the character "a", and so on, until the last "t". Each move is implemented through the m_next array of the current node, which contains a pointer to the next character. At the end of each key value, if this key value represents a complete Tensor name, the m_data of the last character node will not be NULL, it will point to the actual data of the Tensor.

[0085] Figure 3Each row in shows a different level of nodes and their possible transitions. Each node may have up to 62 child nodes, but usually only a few of them are actually used, depending on the key values ​​actually stored in the dictionary tree. In the figure, each character of the key is connected to each other through a series of nodes, forming a complete key value path.

[0086] Usually, a complete Trie tree has a large number of nodes. Taking the nodes required by the 291 weight Tensors of the Meta LLaMA model used in the present invention as an example, the number of nodes required to build a complete Trie tree is 330,000. Since the number of nodes to be created in the Trie tree is too large, the present invention proposes a solution for reasonable pruning of the Trie tree. Figure 1 It can be found that the naming of tensors has a certain pattern, which is related to the network structure of the large model. A Transformer model mainly consists of an encoder and a decoder module. Each encoder or decoder module contains multiple Transformer blocks, that is, layers.

[0087] Specifically, Figure 1 A table showing the weight parameters of the artificial intelligence model. The meaning of each column is as follows:

[0088] name: parameter name, referring to the weight parameter of a specific layer or part in the model.

[0089] type: data type, "Q4_0" indicates a 4-bit quantized integer, and "F32" indicates a 32-bit floating point number.

[0090] dims: The number of dimensions of the parameter, "2" means this is a two-dimensional tensor.

[0091] shape: The shape of the tensor. For example, (32000, 4096) means that this tensor has 32000 rows and 4096 columns.

[0092] weight size: The weight of the parameter, which may indicate the size of the memory space required to store the weight, usually in bytes.

[0093] The naming of the name column usually follows the format of "[layer].[sublayer].[function].[parameter type]", which helps to quickly locate specific parameters in the model code and documentation;

[0094] For example, tok_embeddings.weight represents the weight parameter of "token embeddings";

[0095] norm.weight usually refers to the normalized weight parameter of a layer;

[0096] output.weight refers to the weight parameter of the output layer;

[0097] layers.x.attention.wq.weight represents the query weight in the attention mechanism of layer x.

[0098] In the present invention, the tensor named layer.x, where x can take values ​​0-31, has 288 tensors with the same prefix "layer" and suffix "weight". The present invention prunes these two keywords in the Trie. The specific method is to set the number of m_next pointers of each node to 1 instead of the usual 62, and set its m_is_opt to True, that is, to optimize the node to achieve the purpose of pruning the dictionary (Trie) tree. For details, refer to Figure 4 , Figure 4 It shows a pruned dictionary tree for tensors with a specific naming pattern. In this invention, tensor naming follows the pattern of "layer.x", where x ranges from 0 to 31. Therefore, these tensors have a common prefix "layer" and suffix "weight".

[0099] Normally, each dictionary tree node will have 62 possible m_next pointers, corresponding to 26 uppercase letters, 26 lowercase letters and 10 numeric characters. However, in the pruning process of the present invention, the number of m_next pointers of each node representing the "layer" and "weight" keywords is set to 1. This means that for these specific prefixes and suffixes, the path in the dictionary tree is no longer scattered, but becomes a single straight path.

[0100] The m_is_opt field of nodes with a "layer" prefix and a "weight" suffix is ​​set to True, indicating that these nodes are optimization nodes. This indicates that these nodes are specially designed for efficiency and they represent a path that can be compressed because all tensor names of the "layer.x.weight" format share these common paths.

[0101] This pruning method significantly reduces the number of nodes, thus reducing memory usage, and potentially improves search efficiency since the search path becomes more direct and does not require checking for a large number of null pointers.

[0102] Figure 4 Compared to Figure 3 We have seen considerable pruning effects. Figure 1 key value, this method can reduce the number of nodes to 107,000, and the node optimization ratio is 30%.

[0103] The present invention stores Tensor and its data in a pre-opened space. Since Tensor is a structure, alignment rules need to be considered. The present invention aligns all Tensor structures and subsequent Tensor data (Data) to 32 bytes. Figure 5 Describes in detail how the index nodes of the dictionary (Trie) tree are associated with the Tensor, and how the Tensor and its data are placed in memory. The Tensor structure occupies a fixed 28 bytes, so there will be a 4-byte alignment in each tensor structure, and the space after that is the Data of the current Tensor. If the 32-byte alignment is not met after the end of the Data, additional alignment space is also required;

[0104] Specifically, Figure 5 The upper part is a dictionary tree structure, which is used to represent the index of the Tensor. Each node of the dictionary tree represents a character in the Tensor name. The nodes are arranged in layers, and each layer corresponds to a position in the name. In the dictionary tree, the path extends from the root node (usually representing the beginning of the string) down to the leaf node (usually representing the end of the string).

[0105] There is a key area at the top of the tree labeled "layer.0.att.norm.wgt" etc., indicating that these are the names of Tensors.

[0106] Starting from each key name, there is a path leading to different nodes, each node representing a character in the key name.

[0107] Each node may have up to 62 child nodes, which represent the 26 uppercase English letters, 26 lowercase English letters, and 10 numeric characters.

[0108] Figure 5 The figure shows the paths of three key values, which are mapped through the sequence of nodes in the dictionary tree.

[0109] Figure 5 The lower part of the graph represents the actual layout of the Tensor in memory. Here, the memory is visualized as a grid, with each grid representing a certain amount of bytes in memory.

[0110] The "tensor" area represents the storage location of the Tensor structure, which occupies 28 bytes, followed by a 4-byte alignment space to ensure that the entire structure is 32-byte aligned.

[0111] The "data" area represents the subsequent Tensor data. The amount of data may vary, depending on the actual size of the Tensor.

[0112] The "alignment" area indicates the extra space that needs to be filled if the Tensor data does not fill a 32-byte aligned block.

[0113] The lines between the trie and the memory layout indicate the relationship between a specific node in the trie and a specific Tensor structure in memory. For example, a specific Tensor name is indexed in the trie and then pointed to by a line in the memory layout that represents the Tensor structure.

[0114] exist Figure 5 In the lower half of the graph, each Tensor structure is evenly arranged in a row, and each structure consists of multiple units, indicating the bytes they occupy.

[0115] If the length of the Tensor data is not an integer multiple of 32 bytes, there will be additional alignment padding after the Tensor data to ensure that the next Tensor structure or data starts on a 32-byte alignment boundary.

[0116] The indexing method of the dictionary tree is explained as follows using weight.tensor as an example:

[0117] 1 Preprocess the string with "weight.tensor" as the Key value, that is, delete '.', and then the Key value becomes "weighttensor";

[0118] 2. Take the root node as the root node and find the m_next['w'] value of the node, which is also a node containing the m_next[] array, denoted as A.

[0119] 3. Take node A as the base node and find the value of m_next['e'] of the node, which is also a node that contains the m_next[] array, denoted as B.

[0120] 4 And so on, find the 'i''g''h''t''t''e''n''s''o''r' nodes in turn, and in the last 'r' node, read its m_data value, which is the pointer to the tensor data.

[0121] The method of the present invention is explained below with a specific embodiment:

[0122] 1. #include "op_tensor.hpp": All implementation interfaces and structure definitions of the present invention are in the header file, so it is necessary to include op_tensor.hpp in the source file using the method of the present invention.

[0123] #include "op_tensor.hpp"

[0124] 2. Call the following function tensor_ctx_init: The present invention will pre-allocate memory uniformly according to the large model specifications, and the tensor_context structure will manage it uniformly

[0125] tensor_context*tensor_ctx_init(size_t mem_size)

[0126] 3. If you need to reuse the memory space, you can call tensor_ctx_free to clear all counters and point the memory pointer back to the first address of the memory.

[0127] void tensor_ctx_free(tensor_context*ctx)

[0128] 4. When the program ends, call the following interface to release the memory

[0129] void tensor_ctx_delete(tensor_context*ctx)

[0130] 5. Call create_new_tensor to add a new key to the dictionary tree. If it is the first call, a new empty dictionary tree will be created first, and the new key will be added to the root node. At the same time, space will be requested in the memory managed by tensor_context to store the tensor structure and its data.

[0131] tensor <t>*create_new_tensor(tensor_context*ctx,int32_t dim,int32_tshape[],const void*src=NULL)

[0132] 6. Calling the following function with the same name create_new_tensor with an existing Tensor as a parameter will copy and construct a tensor with the same name and the suffix _copy as the key in the dictionary tree, and copy its structure and data to a new tensor in the memory space maintained by tensor_context

[0133] tensor <t>*create_new_tensor(tensor <t>*t)

[0134] The access and search efficiency of Tensor is compared by the following experiments:

[0135] The requirements are as follows:

[0136] Try to build 10K (10*1024) Tensors;

[0137] The Tensor names are "tensor.0.weight,tensor.1.weight,tensor.2.weight...tensor.10239.weight";

[0138] Tensor data consists of 10 one-dimensional INT random numbers;

[0139] Find these 10K Tensors separately and record the search time;

[0140] The experimental hardware environment is AMD EPYC 754364-core CPU.

[0141] 1. Call the above interface provided by the present invention to achieve the above requirements, and the latency is 2.243us;

[0142] 2. Call LLaMA.CPP to implement the Tensor maintenance scheme implemented by the linked list, with a latency of 1803.367us.

[0143] Based on this implementation plan, we can conclude

[0144] 1. The efficiency of the present invention in searching large model Tensors is nearly 900 times faster than that of typical popular solutions (calculated based on the ratio of the delays of the two solutions).

[0145] 2. By pruning the dictionary (Trie) tree, the dictionary (Trie) tree nodes of this implementation are reduced by 30% compared with the non-optimized solution.

[0146] Summary: Compared with linked lists, the Trie tree greatly improves indexing efficiency by 900 times by exchanging space for time (i.e., the Trie tree stores all possible prefixes to avoid character-by-character comparison, thereby speeding up search). At the same time, the present invention also improves the Trie tree and compresses its Node nodes by 30% in space.

[0147] The protection content of the present invention is not limited to the above embodiments. Without departing from the spirit and scope of the present invention, changes and advantages that can be thought of by those skilled in the art are included in the present invention and are protected by the attached claims.< / t> < / t> < / t>

Claims

1. A method for indexing and saving AI large model Tensor based on dictionary tree, characterized in that: The method comprises the following steps: Step 1: Dictionary tree construction: initialize the root node, expand the node character by character, and mark the end point of the key value; Step 2: Preprocessing: Format the input key value and remove unnecessary characters; Step 3: Build an index: Use the hierarchical structure of the dictionary tree to create a path for each unique key value, where each node represents a character in the key value; Step 4: Search level by level: Starting from the root node of the dictionary tree, traverse the child nodes one by one according to each character of the key value; Step 5: Data association: At the node corresponding to the last character of the key value, associate or save a pointer or reference to the actual data.

2. The method according to claim 1, characterized in that In step 1, the construction of the dictionary tree includes the following steps: Step 1.1, use tensor name as the key value of the dictionary tree, and pre-process the key value to retain legal characters; Step 1.2: Build a root node, and based on the root node, expand the child nodes character by character according to the legal characters obtained in step 1.1 to build a complete dictionary tree.

3. The method according to claim 2, characterized in that Step 1.2 also includes the process of pruning the constructed dictionary tree, merging the Tensor names with the same prefix and suffix together to reduce the number of nodes in the dictionary tree; the dictionary tree is pruned by setting the number of m_next pointers of the merged nodes to 1 and setting m_is_opt to True; In the dictionary tree, the weight data is represented by a Tensor structure, which includes the Tensor name m_name, the Tensor dimension m_dim, the shape of the Tensor in each dimension m_shape, the Tensor gradient change m_stride, the number of Tensor elements m_count, and the Tensor specific data pointer m_data.

4. The method according to claim 2, characterized in that The preprocessing in step 1.1 and the preprocessing in step 2 both refer to removing characters other than legal characters from the output tensor name key value. The legal characters are 26 uppercase and lowercase letters and 10 numeric characters, a total of 62 characters.

5. The method according to claim 1, characterized in that In step 3, each unique Tensor name is used as a key value, and a path in the dictionary tree is created according to the order of the characters in the key value. Each node on the path represents a character in the Tensor name, which is used to retrieve and find Tensor information in the dictionary tree.

6. The method according to claim 1, characterized in that In step 4, during the traversal process, each character is used to match the corresponding child node in the dictionary tree; if the character matches successfully, it moves to the corresponding child node and continues to match the next character until the complete key value is matched or the match cannot continue.

7. The method according to claim 1, characterized in that In step 5, when step 4 is matched successfully and the destination node of the target key value is found, a pointer or reference pointing to the actual data of the Tensor is associated or saved in the destination node, and information or data including the weight Tensor or the activation value Tensor is located; The Tensor data is stored in pre-allocated memory space, and the size of the Tensor structure is ensured to be consistent with the memory alignment requirements.

8. Application of the method as described in any one of claims 1 to 7 in deep learning model reasoning and deployment, model pruning and optimization, and model interpretation.

9. A hardware system for implementing the method according to any one of claims 1 to 7, characterized in that: The hardware system comprises: a memory and a processor; a computer program is stored in the memory, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images