Data processing system, method, device, medium and program product

Abstract

The present application discloses a data processing system, method, device, medium, and program product in the field of computer technology. In this solution, the encoding and decoding of the model inference request are completed on the first node and the second node respectively, and by controlling the scheduling of the nodes, the non-interference and scheduling matching of the encoding and decoding are guaranteed, which can save the time of the model inference process and achieve faster inference response. To a certain extent, it can improve the processing efficiency of the model inference request; further, because the first node caches part of the tree structure (i.e., part of the character features and corresponding key-value data that have been processed), the amount of cached data and excessive cache occupancy during the decoding stage are reduced, which can improve the efficiency of model inference.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing system, method, device, medium and program product. Background Art

[0002] Currently, during the inference process of natural language processing models, the key-value matrix calculated based on the input data needs to be cached. The cache size of the key-value matrix grows infinitely with the context length, resulting in inefficient memory allocation. Frequent memory access also increases inference latency.

[0003] Therefore, how to improve the efficiency of model reasoning is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a data processing system, method, device, medium and program product to improve the efficiency of model reasoning.

[0005] In the first aspect, the present application provides a data processing system, comprising: a control node, and M first nodes and N second nodes deployed according to a set ratio; the control node is used to: obtain multiple model inference requests, and distribute the multiple model inference requests to the M first nodes; any first node is used to: character encode the received model inference request to obtain an encoding result set; match the encoding result set with its own pre-stored N partial tree structures respectively to obtain N matching results; send the N matching results to the control node; the N partial tree structures correspond one-to-one to the N second nodes; the control node is used to: determine the target node in the N second nodes based on the received matching results and the N remaining computing memory space in the N second nodes; the target node is used to: obtain the encoding result set and its corresponding key-value data set, decode the encoding result set and the key-value data set to obtain the corresponding model inference result, and construct a leaf node based on the encoding result set and the key-value data set, and fill the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

[0006] In the second aspect, the present application provides a data processing method, including: a control node in a data processing system obtains multiple model inference requests and distributes the multiple model inference requests to M first nodes; the data processing system includes: a control node, and M first nodes and N second nodes deployed according to a set ratio; any first node in the data processing system character encodes the received model inference request to obtain an encoding result set; matches the encoding result set with its own pre-stored N partial tree structures respectively to obtain N matching results; sends the N matching results to the control node; the N partial tree structures correspond one-to-one to the N second nodes; the control node determines the target node among the N second nodes based on the received matching results and the N remaining computing memory space in the N second nodes, so that the target node obtains the encoding result set and its corresponding key-value data set, decodes the encoding result set and the key-value data set to obtain the corresponding model inference result, and constructs a leaf node based on the encoding result set and the key-value data set, and fills the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

[0007] In a third aspect, the present application provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the aforementioned disclosed data processing method.

[0008] In a fourth aspect, the present application provides a non-volatile storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed data processing method when executed by a processor.

[0009] In a fifth aspect, the present application provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0010] From the above scheme, it can be seen that the present application provides a data processing system, including: a control node, and M first nodes and N second nodes deployed according to a set ratio; the control node is used to: obtain multiple model inference requests, and distribute the multiple model inference requests to M first nodes; any first node is used to: character encode the received model inference request to obtain an encoding result set; match the encoding result set with its own pre-stored N partial tree structures respectively to obtain N matching results; send the N matching results to the control node; the N partial tree structures correspond one-to-one to the N second nodes; the control node is used to: determine the target node in the N second nodes based on the received matching results and the N remaining computing memory space in the N second nodes; the target node is used to: obtain the encoding result set and its corresponding key-value data set, decode the encoding result set and the key-value data set to obtain the corresponding model inference result, and construct a leaf node based on the encoding result set and the key-value data set, and fill the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

[0011] As can be seen, the beneficial effects of this application are as follows: in the data processing system, the encoding and decoding of model inference requests are completed on the first node and the second node respectively. By controlling the scheduling of the nodes, the encoding and decoding are guaranteed to be non-interfering and scheduling matching, which can save time in the model inference process and achieve faster inference response. To a certain extent, it can improve the processing efficiency of model inference requests. Furthermore, because the first node caches a portion of the tree structure (i.e., the processed portion of the character features and the corresponding key-value data), the amount of cached data and excessive cache occupancy during the decoding phase are reduced, which can improve the efficiency of model inference.

[0012] Correspondingly, the data processing method, device, medium and program product provided by this application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0014] Figure 1 A schematic diagram of a data processing system disclosed in this application;

[0015] Figure 2 A schematic diagram of a tree structure disclosed in this application;

[0016] Figure 3A flow chart of a data processing method disclosed in this application;

[0017] Figure 4 A flow chart of another data processing method disclosed in this application;

[0018] Figure 5 A schematic diagram of an electronic device disclosed in this application;

[0019] Figure 6 A server structure diagram provided for this application;

[0020] Figure 7 This is a terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0021] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0022] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0023] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0024] A natural language processing model can be a neural network based on a Transformer. The Transformer structure consists of two main parts: an encoder, which encodes text information, and a decoder, which decodes the encoded text information and outputs the corresponding text. In a decoder-based natural language processing model, the inference generation process consists of two main steps: encoding the current user input, or prompt, often referred to as prefill; and decoding the encoded prompt, often referred to as decoding. Prefill and decoding correspond to the P and D nodes, respectively, described below. Prefill calculates the corresponding QKV matrix, while decoding is performed during decoding.

[0025] Currently, natural language processing models require caching the key-value matrix (KV) calculated based on input data during inference. The cache size of the KV matrix grows infinitely with context length, leading to inefficient memory allocation. Frequent memory access also increases inference latency. To address this issue, this application provides a data processing solution that improves model inference efficiency.

[0026] See also Figure 1 As shown, an embodiment of the present application discloses a data processing system, including: a control node, and M first nodes and N second nodes deployed in a set ratio. There can be multiple control nodes. When there are multiple control nodes, the scheduling of control rights can be implemented by referring to dual-control and multi-control distributed systems.

[0027] The default ratio of M to N is 1:1, which can be adjusted flexibly if needed. The control node, first node, and second node can all be servers and include a CPU and a GPU. Accordingly, the CPU is the controller processor, and the CPU memory is the control memory; the GPU is the computation processor, and the GPU memory is the computation memory. Of course, the GPU can also be replaced with other types of acceleration devices, such as FPGAs.

[0028] In this embodiment, the control node is used to obtain multiple model inference requests and distribute the multiple model inference requests to M first nodes.

[0029] Any first node is used for: performing character encoding on the received model inference request to obtain an encoded result set; respectively matching the encoded result set with N partial tree structures pre-stored by itself to obtain N matching results; sending the N matching results to the control node; the N partial tree structures correspond to N second nodes one by one. That is to say, the N second nodes synchronize the parts of the global tree structure they store to each first node, so that each first node stores N partial tree structures. The leaf nodes of the global tree structure store the key-value data KV matrix corresponding to a character feature token that has been processed, in the form of a key-value pair. Thus, for the same character feature token in the future, the calculation process of the KV matrix can be omitted. Among them, the first node can calculate the corresponding attribute matrix based on a token, including: the first matrix Q, the second matrix K, and the third matrix V, to obtain the QKV matrix; the Q matrix contains the corresponding information of the Q of the previous token of the current token, so only one Q is cached; while the KV matrices of different tokens need to be cached in their entirety.

[0030] It should be noted that the token can be the encoding of each character in the model inference request. For example: the model inference request is: Please generate a movie review for movie S. Then the character feature tokens corresponding to this request can include: Please, for, movie S, generate, a movie review, and the encodings corresponding to these five words are recorded as a branch in the tree structure in the original word order. For details, please refer to Figure 2 . Of course, it is also possible to perform word segmentation and encoding on each Chinese character. Then the character feature tokens corresponding to this request can include: Please, needle, pair, electricity, shadow, S, generate, become, one, section, shadow, review, and the encodings corresponding to these twelve words; the specific word segmentation method can be flexibly determined based on the actual situation. It can be seen that an encoded result set of a model inference request includes multiple character features, and each character feature token corresponds to a key-value data (i.e., the KV matrix), and these key-value data constitute a key-value data set.

[0031] Correspondingly, respectively matching the encoded result set with the N partial tree structures pre-stored by itself means: matching all the character feature tokens corresponding to the same model inference request, in their arranged order, with the branches of the N partial tree structures. Please refer to Figure 2 , if the model inference request is: Please generate a book review for book A, then this request can be Figure 2In the matching result, the corresponding matching result (i.e., matching ratio) is 2 / 5, where 2 is the number of matched tokens and 5 is the total number of tokens in the encoding result set corresponding to the current model inference request, that is, the number of token segments in the current model inference request. Therefore, any first node is used to determine multiple character features in the encoding result set arranged in the first order, calculate the character proportions in the N partial tree structures in the first order, and use the obtained N character proportions as the matching ratio in the N matching results. Therefore, the matching process must be performed according to the word order of each token in the request.

[0032] The control node is used to determine the target node among the N second nodes based on the received matching results and the N remaining spaces of the operational memory among the N second nodes. Specifically, the first node can send the matching results that exceed the first threshold among the N matching results to the control node. That is, the first node informs the control node of the second node with a higher matching rate, so that the control node selects the second node for performing the decoding operation for the currently processed model inference request based on this and in combination with the N remaining spaces of the operational memory among the N second nodes. The selection goal is that the remaining space of the operational memory in the second node is large enough, and the matching rate of the partial tree structure corresponding to the current second node and the encoding result set is high enough.

[0033] It should be noted that, in order to enable the control node to know the correspondence between each matching result and the second node, each matching result includes not only the matching rate, but also the identification information of the second node corresponding to the partial tree structure corresponding to the matching rate.

[0034] In one embodiment, the control node selects the target node in the following manner: detect the difference between the received matching results; if the difference does not exceed the second threshold, then select the second node based on the matching rate to perform the currently requested decoding operation. The computational complexity and computational amount are similar, then the second node is selected based on the remaining space of the computing memory, that is, the second node with the largest remaining space of the computing memory is selected as the target node among the N second nodes; if the difference exceeds the second threshold, then the second node with a high matching rate is selected to reduce the computational complexity and computational amount, and therefore the maximum matching result is selected among the received matching results, and the second node corresponding to the maximum matching result (that is, the matching result with the largest matching rate) is selected as the target node.

[0035] The target node is used to obtain the encoding result set and its corresponding key-value data set, decode the encoding result set and the key-value data set to obtain the corresponding model inference result, and construct leaf nodes based on the encoding result set and the key-value data set, and fill the leaf nodes into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

[0036] In this embodiment, the process of storing the partial tree structure in each first node includes: the target node extracts a preset number of tree branches from the root node of the global tree structure; synchronizes the extracted branches to M first nodes; and, accordingly, the M first nodes are configured to update the partial tree structure corresponding to the target node stored in their own storage based on the received branches. If the global tree structure in any second node is updated, the corresponding updated content is also synchronized to all first nodes.

[0037] It should be noted that, since the leaf nodes of the global tree structure and the partial tree structure are in the form of key-value pairs, there is a key-value data KV matrix corresponding to a character feature token that has been processed, and this token and this KV matrix are both calculated by the first node, that is, the first node already has the token and KV matrix. In order to reduce the amount of transmission of tree structure-related data between the first node and the second node, the second node can only send the corresponding token to each first node, so that the corresponding first node can query the corresponding KV matrix in itself based on the received token, and complete the construction of the leaf node and the update of the corresponding partial tree structure. Therefore, in one embodiment, the target node is used to: send the character features in the intercepted branches to M first nodes; accordingly, the M first nodes are used to: match the corresponding key-value data according to the received character features, and update the partial tree structure corresponding to the target node pre-stored in itself according to the received character features and the matched corresponding key-value data.

[0038] It should also be noted that, since the tree structure stored in the second node is larger than the storage of the first node, the first node can also send its own unmatched tokens to the second node used to complete the same model inference request, so that the second node performs the matching operation to further reduce the computational complexity and amount of the first node for KV. Therefore, in one embodiment, any first node is also used to: send the character features in the encoding result set that are not matched in the partial tree structure corresponding to the target node to the target node; accordingly, the target node is used to: match the received character features with the global tree structure in itself, and return the unmatched character features to the corresponding first node, so that the first node calculates the key-value data corresponding to the unmatched character features.

[0039] Since the control node, the first node, and the second node can all be servers, they can all include: a CPU and a GPU; accordingly, the CPU is the controller processor, and the CPU memory is the control memory; the GPU is the computing processor, and the GPU video memory is the computing memory. Then in one embodiment, any first node and any second node include: a control processor and a computing processor, the processing efficiency of the control processor is lower than the processing efficiency of the computing processor; the data access speed of the control memory of the control processor is lower than the data access speed of the computing memory of the computing processor. Accordingly, any first node is used to: use the computing processor in itself to perform character encoding operations, so that character encoding is faster; use the control processor in itself to perform matching operations, the matching operation is not complex, and is performed by the control processor with lower computing efficiency, which can save computing processor resources; accordingly, the target node is used to: use the computing processor in itself to perform decoding operations; use the control processor in itself to perform tree structure filling operations.

[0040] In one embodiment, the control memory of the control processor in any first node is used to: store N partial tree structures to avoid occupying the operation memory space; the operation memory of the operation processor in any first node is used to: load the character encoding operation related data and key value data calculation operation related data currently being executed by the operation processor for use by the operation processor; accordingly, the control memory of the control processor in any second node is used to: store the corresponding global tree structure and the decoding operation related data to be executed, that is, the pending tasks waiting for the operation processor to execute; the operation memory of the operation processor in any second node is used to: load the decoding operation related data currently being executed by the operation processor for use by the operation processor.

[0041] To conserve computational memory in the first node, the calculated QKV matrix in the computational memory can be migrated to the control memory of the first node. To avoid control memory fragmentation and improve its utilization, a first mapping table is constructed for the control memory of the first node. The first mapping table records the mapping relationship between consecutive virtual addresses and physical addresses of the control memory. Data to be written to the control memory (e.g., the QKV matrix) can be used to fill fragmented locations in the control memory based on this mapping relationship. In one embodiment, any first node is configured to: according to the first mapping table, migrate the first matrix Q, the second matrix K, and the third matrix V in the computational memory of its own computational processor to the fragmented locations of the control memory of its own control processor, with a first predetermined size; wherein the control memory of any first node is divided into multiple memory areas with the first predetermined size, and the first mapping table records the virtual address segments corresponding to each memory area.

[0042] Based on the same idea, a second mapping table is constructed for the operation memory in the second node. The second mapping table records the mapping relationship between continuous virtual addresses and physical addresses of the operation memory. According to this mapping relationship, the data to be written into the operation memory (such as: the Q matrix updated locally in the operation memory, the decoding operation related data read from the control memory of the same node) can be filled into the fragmented position in the operation memory. In one embodiment, any second node is used to: update the first matrix Q in the operation memory of its own operation processor according to the second mapping table with a second set size; wherein the operation memory in any second node is divided into multiple memory areas with a second set size, and the second mapping table records the virtual address segment corresponding to each memory area. Accordingly, the operation memory of the operation processor in any second node is used to: load the decoding operation related data currently being executed by the current operation processor into the fragmented position of the current operation memory according to the second mapping table with a second set size.

[0043] In order to achieve orderly processing of each model inference request in the first node and the second node, so that the waiting time of each model inference request is not too long, the model inference request can be paused or other scheduling processing according to the priority of the model inference request. In one embodiment, any first node and any second node are used to: if any model inference request A times out for the first time, then pause the model inference request B that is being executed in the current node, has the latest arrival time and the lowest priority, and migrate the relevant data of the paused model inference request B from the operation memory of the operation processor in the current node to the control memory of the control processor in the current node, so that the paused model inference request B enters a waiting state; then use the operation processor in the current node to execute the model inference request A that has timed out for the first time, so that A does not have to wait too long; if the waiting of the suspended model inference request B times out, use the operation processor in the current node to execute the suspended model inference request B, so that B does not have to wait too long.

[0044] It should be noted that the corresponding tree structures in the first node and the second node can be updated in a timely manner as follows. In the first node and the second node, the tree structure performs branch deletion according to its matching rate. In one embodiment, any first node is used to: if there is a branch with a matching rate lower than the third threshold in any part of the tree structure, then delete the branch; if there is a branch with a matching rate not lower than the third threshold in any part of the tree structure, then when the matching rate of the branch in the corresponding second node is lower than the fourth threshold, the branch is deleted synchronously in the current second node and the current first node; if the branch is being processed, the branch is deleted after the processing is completed; if the matching rate of the branch in the corresponding second node is not lower than the fourth threshold and the tree structure data loaded in the operation memory of the current second node (data used for the decoding operation currently being performed) exceeds the preset video memory threshold, then according to the matching rate and usage rate, the tree structure data loaded in the operation memory of the current second node is deleted. For example, if the matching rate of a branch in the second node reaches the threshold dt1, and the amount of tree structure data occupied in the computational memory exceeds the preset value C of the video memory, the minimum utilization principle is adopted to eliminate branches with low matching rates in ascending order.

[0045] Furthermore, the first node and the second node can change node types with each other. For example: the first node is changed to the second node, and the second node is changed to the first node. In one embodiment, any first node and any second node are used to: if the load of the current node is lower than the preset load threshold and the throughput of the current node is lower than the threshold throughput threshold, or if the load fluctuation of the current node is greater than the preset fluctuation threshold, then when the current node is the first node, the current node is changed to the second node; when the current node is the second node, the current node is changed to the first node. Among them, when changing the node type, you can wait for the normal running and waiting to run tasks in the node of the changed type to be completed, and then no longer receive new tasks, then change the node type, and then receive new tasks; or, if the remaining GPU resources (GPU remaining video memory and / or GPU remaining computing power) of other nodes of the same type of node of the changed type can complete the unfinished tasks of the node of the changed type, then the normal running and waiting to run tasks in the node of the changed type will be arranged from large to small according to the remaining GPU resources of other nodes of the same type, and migrated to other suitable nodes of the same type for processing.

[0046] In one embodiment, the control node is used to: if the matching rate of the global tree structure in at least two second nodes is lower than a preset fifth threshold, merge the global tree structures in at least two second nodes into the same second node in at least two second nodes, and determine that there is no second node of the global tree structure in at least two second nodes, and change the second node to the first node; wherein the merged global tree structure is not greater than a preset sixth threshold, and the merged global tree structure cannot exceed the storage area (i.e., the first area) allocated for it by the control memory for storing it.

[0047] In one embodiment, any first node and any second node are configured to determine whether to change the node type of the current node based on the number of highest-priority requests among all waiting requests at the current node and the throughput of the current node if the number of waiting requests at the current node exceeds a preset waiting task quantity. If the number of highest-priority requests among all waiting requests at the current node exceeds a preset upper limit and the throughput of the current node is lower than a preset lower limit, indicating that the highest-priority requests can no longer wait to be processed, the node type of the current node is changed; otherwise, the node type of the current node may remain unchanged.

[0048] As can be seen, in the data processing system of this embodiment, encoding and decoding of model inference requests are completed on the first and second nodes, respectively. By controlling node scheduling, encoding and decoding are guaranteed to be independent of each other and to be well-matched, thus saving time during the model inference process and achieving faster inference responses. This improves the processing efficiency of model inference requests to a certain extent. Furthermore, because the first node caches a portion of the tree structure (i.e., processed character features and corresponding key-value data), the amount of cached data and excessive cache usage during the decoding phase are reduced, thereby improving model inference efficiency.

[0049] See Figure 3 , a data processing process in a data processing system includes:

[0050] Step 1: Before inference begins, determine the ratio of P nodes (first nodes) to D nodes (second nodes) and complete system initialization configuration.

[0051] The default PD ratio is 1:1. If other ratios are set in the system, this ratio can also be used.

[0052] Nodes in the data processing system are divided into two categories: control nodes, which are used to adjust the PD ratio and assign inference tasks; and compute nodes, which are used for specific PD calculations. Control nodes dynamically adjust the PD ratio by monitoring information such as the token generation latency of compute nodes. Furthermore, control nodes assign the P node's model inference request to a D node based on the matching rate returned by the P node in the compute node.

[0053] Step-2: Matching of P nodes and D nodes.

[0054] When a model inference request reaches a P node, it begins computing, encoding the prompt in the request into a corresponding token. A single model inference request may be encoded into multiple tokens. After encoding, a matching operation is performed to minimize wasted computing resources. This matching operation determines which tokens have already had their KVs calculated and which have not. Subsequently, only those tokens for which KVs have not been calculated are then KVs calculated. This matching operation also assigns the model inference request to a D node for decoding and generation.

[0055] In this embodiment, the RadixTree data structure is used to construct a tree structure, and the tokens whose KVs have been calculated and their KVs are sequentially constructed into a tree structure. Each D node stores the global tree structure of all tokens it has processed, and each P node stores a part of the global tree structure in each D node, and the first 10 layers can generally be used as this part. Since both the P node and the D node include a CPU and a GPU, the global tree structure can be stored in the CPU memory for the D node, and all KVs of the global tree structure can be stored in the GPU memory to save memory space. At the same time, if there is an update to the global tree structure in the D node (addition or deletion of a node), the tree structure in the P node is updated accordingly. The P node stores the tree structure in the CPU memory. The leaf nodes of the tree structure use the token as the key and the corresponding KV as the value.

[0056] In P-node encoding, in order to reduce resource waste, each time a QKV of a fixed size is generated, the QKV will be copied to the CPU memory to reduce large-block data transmission and video memory occupancy. Specifically, the video memory of the P node is physically divided into blocks of fixed size. When a block of video memory is filled, it will be transferred to the CPU memory. The CPU memory maps the CPU memory physical address to a virtual address and establishes a first mapping table. The continuous virtual addresses recorded in the first mapping table may not be continuous physical addresses in the CPU memory. The CPU memory is also divided into corresponding block sizes according to the block size divided by the GPU, and the mapping table is used to match the virtual address block with the actual physical address block.

[0057] The aforementioned matching operation is specifically as follows: in the P node, the token generated by the same model inference request is matched with the token of the leaf node of the tree structure stored in the CPU memory to obtain the matching ratio, and then the matching ratio that exceeds the preset matching threshold, such as 30%, is selected and sent to the control node so that the control node knows which second node can be selected. After that, the control node combines the matching ratio exceeding the threshold sent by the P node and the remaining video memory capacity fed back by the corresponding D node to perform further task assignment. The specific process is: among the nodes with higher matching rates, if the matching degree of these nodes does not differ by more than 10%, priority is given to sending this model inference request to the second node with the larger remaining video memory for execution. If it exceeds 10%, the second node with the highest matching ratio is designated to run this task.

[0058] Step-3: Matching and decoding operations of D node.

[0059] After determining the D node, the first thing to do is to send the tokens that the P node has not matched to the D node, so that D can continue to find out which tokens among the unmatched tokens have already calculated the KV and which have not. Then the D node will send the uncalculated tokens to P, and let P calculate the QKV for them. After P completes the calculation, it sends the QKV to D. The D node loads the KV corresponding to the matched token into the GPU memory and reserves memory space for the QKV calculated by P. If the memory space is insufficient, the loadable part will be loaded into the GPU memory first, and a separate queue will be maintained on the CPU memory. The queue stores the decoding-related data that cannot be loaded temporarily due to insufficient memory. Among them, the CPU memory of the D node is divided into two areas, one is to store the global tree structure, referred to as the first area; the other is to store the area that cannot be loaded into the GPU memory to perform the generation task, referred to as the second area, which is the aforementioned queue.

[0060] Because the generation of the Q matrix depends on all the aforementioned Qs, the D node must always store the latest Q in the video memory. To prevent GPU memory resource constraints, a second mapping table is constructed for the D node's GPU memory. This table stores each newly generated Q using the same principles as the first mapping table. After the D node outputs an answer to a model inference request, the relevant data for that request is moved from the GPU memory to the CPU memory or output to the user end to reduce video memory usage. The relevant data for the new decoding operation can then be loaded into the freed-up fragmented locations in the video memory according to the second mapping table, improving video memory utilization and reducing fragmentation.

[0061] Accordingly, data migration between GPU memory and CPU memory in P and D nodes can also include scheduling and executing inference requests of different priorities. Assume that the request priority is set to three levels: 0, 1, and 2, with increasing importance. If model inference request A originally reaches the preset highest priority, it will be set to 2. When model inference request A enters node D, its priority remains at 2. When a new highest-level inference request B arrives, A will give way to B, allowing B to execute. For example, if a model inference request A has not yet begun executing after waiting for the preset waiting time T1, the most recently executed, lowest-priority request B will be evicted to the second area of CPU memory for temporary storage, allowing model inference request A to begin computation on the GPU. If the evicted model inference request B waits for more than the preset waiting time T2, where T2 is greater than T1, model inference request B will be placed on the GPU for further computation. If there are multiple such model inference requests, they will be sorted by the time they arrived at T2.

[0062] Furthermore, in the P node and the D node, the tree structure update can be implemented by referring to the following process. In the P node, if the matching rate of a branch of the tree structure does not reach the preset value pt1 within a period of time, the branch will be deleted; if the branch is in use, it will be deleted after use. If it exceeds the preset value pt1, it will be updated in combination with the matching rate of the corresponding global tree in the D node. Specifically: (1) If the matching rate of a branch of the global tree in the D node does not reach the threshold dt1 within a certain period of time, the branch will be removed and the corresponding tree structure in the P node will be updated synchronously; (2) If the matching rate of a branch of the global tree in the D node reaches the threshold dt1 within a certain period of time, and the memory usage in the RadixTree exceeds the preset memory preset value C, the minimum utilization principle will be adopted to remove branches in the order of matching rate from small to large (removed below C).

[0063] Step-4: Dynamic real-time change of PD node type.

[0064] Considering that the computing bottlenecks of P nodes and D nodes are different, when performance degradation occurs on one side, the node type is adjusted to alleviate the performance degradation problem.

[0065] Specifically, observe the load operation of P nodes and D nodes. If the number of tasks loaded on a node in the past period is less than 80% of the average in its forward time window, and the throughput is less than 80% of the average throughput in the forward time window, it means that the node is not busy yet and the node type can be changed. Alternatively, if a node has experienced significant load changes in the past period, such as large fluctuations in GPU utilization, it means that the tasks for the node are not stable. Therefore, in order to better utilize computing resources, the node type can be changed.

[0066] For the D node, if the tree structure matching rate of more than or equal to 2 nodes is continuously lower than the threshold pt1, it means that the tasks running on these nodes can be merged, then the global trees therein are merged, and the merging rules are as follows: according to the size of the global trees in these nodes, merge them in order from small to large. After each merge, determine whether the merged tree structure exceeds the first area size of the CPU. If it exceeds, the merge will not continue, and the tree obtained by the above merger will be stored in a node. The remaining empty nodes are marked as nodes of changeable type; then continue with the trees to be merged of other nodes. If it does not exceed, continue merging until it exceeds the first area size of the CPU. In one example, if the size of the global tree in the D node exceeds the threshold, the least commonly used token and the corresponding KV Cache in the global tree can be deleted according to the principle of minimum utilization.

[0067] PD node type swapping can also be achieved as follows: Taking P nodes as an example, if more than 20% of the preset number of waiting tasks remain uncompleted, some D nodes with lower throughput will be replaced with P nodes. Taking D nodes as an example, if more than 20% of the preset number of waiting tasks remain uncompleted and the number of high-priority tasks currently stored in the second area exceeds a preset value, some P nodes with lower throughput will be replaced with D nodes.

[0068] In this embodiment, PD separation saves model inference time. The RadixTree-based matching and allocation mechanism enables the model to achieve faster inference speed and more appropriate inference matching based on the remaining GPU memory capacity and cache matching rate before inference, reducing the waste of computing resources. A mapping table maps physical addresses to virtual addresses, and maps discontinuous physical memory / main memory to continuous virtual memory / main memory, reducing resource waste and improving memory / main memory utilization. Real-time changes in PD types can reduce inference time.

[0069] The following introduces a data processing method provided in an embodiment of the present application. The data processing method described below can be referenced with other embodiments described in this document.

[0070] See also Figure 4As shown, the embodiment of the present application discloses a data processing method, including:

[0071] S401. The control node in the data processing system obtains multiple model inference requests and distributes the multiple model inference requests to M first nodes; the data processing system includes: a control node, and M first nodes and N second nodes deployed according to a set ratio.

[0072] S402. Any first node in the data processing system character encodes the received model inference request to obtain an encoding result set; matches the encoding result set with its own N pre-stored partial tree structures to obtain N matching results; sends the N matching results to the control node; the N partial tree structures correspond one-to-one to the N second nodes.

[0073] S403. The control node determines a target node among the N second nodes based on the received matching results and the remaining space of the N computing memories among the N second nodes, so that the target node obtains the encoding result set and its corresponding key-value data set, decodes the encoding result set and the key-value data set to obtain the corresponding model inference result, and constructs a leaf node based on the encoding result set and the key-value data set, and fills the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

[0074] In one embodiment, any first node is used to: determine multiple character features arranged in a first order in the encoding result set, calculate the character proportions in the N partial tree structures according to the first order, and use the obtained N character proportions as the matching rate in the N matching results.

[0075] In one embodiment, any first node is configured to send a matching result exceeding a first threshold among the N matching results to a control node.

[0076] In one embodiment, the control node is used to: detect the difference between the received matching results; if the difference does not exceed a second threshold, select the second node with the largest remaining space in the computing memory among the N second nodes as the target node; if the difference exceeds the second threshold, select the maximum matching result among the received matching results, and use the second node corresponding to the maximum matching result as the target node.

[0077] In one embodiment, the target node is used to: start from the root node of the global tree structure, intercept a preset number of levels of branches; synchronize the intercepted branches to M first nodes; accordingly, the M first nodes are used to: update the partial tree structure corresponding to the target node pre-stored in themselves according to the received branches.

[0078] In one embodiment, the target node is used to send the character features in the intercepted branches to M first nodes; accordingly, the M first nodes are used to match the corresponding key-value data according to the received character features, and update the partial tree structure corresponding to the target node pre-stored in themselves according to the received character features and the matched corresponding key-value data.

[0079] In one embodiment, any first node is also used to: send character features in the encoding result set that are not matched in the partial tree structure corresponding to the target node to the target node; accordingly, the target node is used to: match the received character features with the global tree structure in itself, and return the unmatched character features to the corresponding first node, so that the first node calculates the key value data corresponding to the unmatched character features.

[0080] In one embodiment, any first node and any second node include: a control processor and an operation processor; accordingly, any first node is used to: use the operation processor in itself to perform character encoding operations; use the control processor in itself to perform matching operations; accordingly, the target node is used to: use the operation processor in itself to perform decoding operations; use the control processor in itself to perform tree structure filling operations.

[0081] In one embodiment, the control memory of the control processor in any first node is used to: store N partial tree structures; the operation memory of the operation processor in any first node is used to: load data related to the character encoding operation and key value data calculation operation being executed by the current operation processor; the control memory of the control processor in any second node is used to: store the corresponding global tree structure and data related to the decoding operation to be executed; the operation memory of the operation processor in any second node is used to: load data related to the decoding operation being executed by the current operation processor.

[0082] In one embodiment, any first node is used to: according to a first mapping table, migrate the first matrix, the second matrix and the third matrix in the operation memory of its own operation processor to the fragmented position of the control memory of its own control processor with a first set size; wherein the control memory in any first node is divided into multiple memory areas with the first set size, and the first mapping table records the virtual address segment corresponding to each memory area.

[0083] In one embodiment, any second node is used to: update the first matrix in the operation memory of its own operation processor according to the second mapping table with a second set size; wherein the operation memory in any second node is divided into multiple memory areas with the second set size, and the second mapping table records the virtual address segment corresponding to each memory area.

[0084] In one embodiment, the operation memory of the operation processor in any second node is used to load data related to the decoding operation being executed by the current operation processor into a fragmented position of the current operation memory with a second set size according to the second mapping table.

[0085] In one embodiment, any first node and any second node are used to: if any model inference request times out for the first time, suspend the model inference request that is being executed in the current node, has the most recent arrival time and the lowest priority, and migrate the relevant data of the suspended model inference request from the operation memory of the operation processor in the current node to the control memory of the control processor in the current node, so that the suspended model inference request enters a waiting state; then use the operation processor in the current node to execute the model inference request that has timed out for the first time; if the waiting of the suspended model inference request times out, use the operation processor in the current node to execute the suspended model inference request.

[0086] In one embodiment, any first node is used to: if there is a branch in any partial tree structure whose matching rate is lower than a third threshold, delete the branch; if there is a branch in any partial tree structure whose matching rate is not lower than the third threshold, then when the matching rate of the branch in the corresponding second node is lower than the fourth threshold, delete the branch simultaneously in the current second node and the current first node; if the branch is being processed, delete the branch after the processing is completed; if the matching rate of the branch in the corresponding second node is not lower than the fourth threshold and the tree structure data loaded in the computing memory of the current second node exceeds the preset video memory threshold, then the tree structure data loaded in the computing memory of the current second node is deleted according to the matching rate and usage rate.

[0087] In one embodiment, any first node and any second node are used: if the load of the current node is lower than a preset load threshold and the throughput of the current node is lower than a threshold throughput threshold, or if the load fluctuation of the current node is greater than a preset fluctuation threshold, then when the current node is the first node, the current node is changed to the second node; when the current node is the second node, the current node is changed to the first node.

[0088] In one embodiment, the control node is used to: if the matching rate of the global tree structure in at least two second nodes is lower than a preset fifth threshold, merge the global tree structures in at least two second nodes into the same second node in at least two second nodes, and determine that there is no second node of the global tree structure in at least two second nodes, and change the second node to the first node; wherein the global tree structure obtained by the merger is not greater than the preset sixth threshold.

[0089] In one embodiment, any first node and any second node are used to: if the request waiting amount in the current node is higher than the preset waiting task amount, determine whether to change the node type of the current node based on the number of highest priority requests among all waiting requests in the current node and the throughput of the current node.

[0090] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0091] As can be seen, this embodiment saves model inference time based on PD separation. The RadixTree-based matching and allocation mechanism enables the model to achieve faster inference speed and more appropriate inference matching based on the remaining GPU memory capacity and cache matching rate before inference, reducing the waste of computing resources. By mapping physical addresses to virtual addresses through a mapping table, discontinuous physical memory / main memory is mapped to continuous virtual memory / main memory, reducing resource waste and improving memory / main memory utilization. Real-time changes in PD types can reduce inference time.

[0092] The following describes an electronic device provided in an embodiment of the present application, and the electronic device described below can be referenced with other embodiments described herein. The electronic device can be a control node, a first node, or a second node.

[0093] See also Figure 5 As shown, an embodiment of the present application discloses an electronic device, including: a memory 501 for storing a computer program; a processor 502 for executing the computer program to implement the method disclosed in any of the above embodiments.

[0094] Furthermore, the embodiment of the present application also provides an electronic device. The electronic device can be Figure 6 The server shown can also be Figure 7 The terminal shown. Figure 6 and Figure 7 Each of the diagrams is a structural diagram of an electronic device according to an exemplary embodiment, and the contents in the diagrams cannot be considered as any limitation on the scope of use of the present application.

[0095] Figure 6 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store a computer program, which is loaded and executed by the processor to implement the relevant steps of the data processing disclosed in any of the aforementioned embodiments.

[0096] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface is used to obtain external input data or output data to the outside world. The specific interface type can be selected according to specific application needs and is not specifically limited here.

[0097] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer programs and data, etc. The storage method can be temporary storage or permanent storage.

[0098] The operating system is used to manage and control the hardware devices and computer programs on the server, enabling the processor to operate and process data in the memory. It can be Windows Server, NetWare, Unix, Linux, etc. In addition to computer programs capable of performing the data processing methods disclosed in any of the aforementioned embodiments, computer programs can also include computer programs capable of performing other specific tasks. Data can include data such as application update information and other data such as application developer information.

[0099] Figure 7 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may specifically include but is not limited to a smartphone, tablet computer, laptop computer or desktop computer.

[0100] Generally, the terminal in this embodiment includes: a processor and a memory.

[0101] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content required to be displayed on the display. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0102] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is used to store at least the following computer program, wherein, after the computer program is loaded and executed by the processor, it can implement the relevant steps in the data processing method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include but is not limited to update information of the application.

[0103] In some embodiments, the terminal may further include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0104] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure.

[0105] A non-volatile storage medium provided in an embodiment of the present application is introduced below. The non-volatile storage medium described below can be referenced with other embodiments described herein.

[0106] A non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the aforementioned embodiment. The non-volatile storage medium is a computer-readable non-volatile storage medium that, as a carrier for resource storage, may be a read-only memory, random access memory, a magnetic disk, or an optical disk. The resources stored thereon include an operating system, a computer program, and data, and the storage method may be either temporary or permanent.

[0107] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0108] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned data processing method when executed by a processor.

[0109] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0110] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0111] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-volatile storage medium known in the art.

[0112] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A data processing system, characterized in that: include: A control node, and M first nodes and N second nodes deployed in a set ratio; The control node is used to: obtain multiple model inference requests and distribute the multiple model inference requests to the M first nodes; Any first node is used to: perform character encoding on the received model inference request to obtain an encoding result set; Matching the encoding result set with the N pre-stored partial tree structures respectively to obtain N matching results; and sending the N matching results to the control node; The N partial tree structures correspond one-to-one to the N second nodes; The control node is configured to: determine a target node among the N second nodes according to the received matching result and the remaining space of N operation memories among the N second nodes; The target node is used to: obtain the encoding result set and its corresponding key-value data set, decode the encoding result set and the key-value data set to obtain the corresponding model inference result, and construct a leaf node based on the encoding result set and the key-value data set, and fill the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

2. The data processing system according to claim 1, wherein: Any first node is used to: determine multiple character features arranged in a first order in the encoding result set, calculate the character proportions in the N partial tree structures according to the first order, and use the obtained N character proportions as the N matching results.

3. The data processing system according to claim 1, wherein: Any first node is configured to send a matching result exceeding a first threshold among the N matching results to the control node.

4. The data processing system according to claim 1, wherein: The control node is configured to: detect a difference between the received matching results; if the difference does not exceed a second threshold, select a second node with the largest remaining space in the computational memory from among the N second nodes as the target node; If the difference exceeds the second threshold, a maximum matching result is selected from the received matching results, and a second node corresponding to the maximum matching result is used as the target node.

5. The data processing system according to claim 1, wherein: The target node is used to: start from the root node of the global tree structure, intercept a preset number of levels of branches; synchronize the intercepted branches to the M first nodes; Correspondingly, the M first nodes are configured to update a partial tree structure corresponding to the target node pre-stored therein according to the received branches.

6. The data processing system according to claim 5, wherein: The target node is used to: send the character features in the intercepted branches to the M first nodes; Correspondingly, the M first nodes are used to match corresponding key-value data according to the received character features, and update the partial tree structure corresponding to the target node pre-stored in the first nodes according to the received character features and the matched corresponding key-value data.

7. The data processing system according to claim 1, wherein: Any first node is further configured to: send character features in the encoding result set that are not matched in the partial tree structure corresponding to the target node to the target node; Accordingly, the target node is used to match the received character features with the global tree structure in itself, and return the unmatched character features to the corresponding first node, so that the first node calculates the key value data corresponding to the unmatched character features.

8. The data processing system according to claim 1, wherein: Any first node and any second node include: a control processor and an operation processor; Accordingly, any first node is configured to: utilize its own operation processor to perform character encoding operations; utilize its own control processor to perform matching operations; Accordingly, the target node is configured to: utilize the arithmetic processor within itself to perform a decoding operation; and utilize the control processor within itself to perform a tree structure filling operation.

9. The data processing system according to claim 8, wherein: The control memory of the control processor in any first node is used to store the N partial tree structures; the operation memory of the operation processor in any first node is used to load the character encoding operation related data and the key value data calculation operation related data currently being executed by the operation processor; The control memory of the control processor in any second node is used to store the corresponding global tree structure and data related to the decoding operation to be executed; the operation memory of the operation processor in any second node is used to load data related to the decoding operation being executed by the current operation processor.

10. The data processing system according to claim 9, wherein: Any first node is used to: according to a first mapping table, migrate the first matrix, the second matrix and the third matrix in the operation memory of its own operation processor to the fragmented position of the control memory of its own control processor with a first set size; wherein the control memory in any first node is divided into multiple memory areas with the first set size, and the first mapping table records the virtual address segment corresponding to each memory area.

11. The data processing system according to claim 9, wherein: Any second node is used to: update the first matrix in the operation memory of its own operation processor according to the second mapping table with a second set size; wherein the operation memory in any second node is divided into multiple memory areas with the second set size, and the second mapping table records the virtual address segment corresponding to each memory area.

12. The data processing system according to claim 11, wherein: The operation memory of the operation processor in any second node is used to: load the decoding operation related data being executed by the current operation processor into the fragmented position of the current operation memory according to the second mapping table and with the second set size.

13. The data processing system according to any one of claims 1 to 11, characterized in that: Any first node and any second node are used to: if any model inference request times out for the first time, suspend the model inference request that is being executed in the current node, has the most recent arrival time and the lowest priority, and migrate the relevant data of the suspended model inference request from the operation memory of the operation processor in the current node to the control memory of the control processor in the current node, so that the suspended model inference request enters a waiting state; then use the operation processor in the current node to execute the model inference request that has timed out for the first time; if the waiting of the suspended model inference request times out, use the operation processor in the current node to execute the suspended model inference request.

14. The data processing system according to any one of claims 1 to 11, characterized in that: Any first node is used to: if there is a branch with a matching rate lower than a third threshold in any partial tree structure, then delete the branch; if there is a branch with a matching rate not lower than the third threshold in any partial tree structure, then when the matching rate of the branch in the corresponding second node is lower than the fourth threshold, then delete the branch simultaneously in the current second node and the current first node; if the branch is being processed, then delete the branch after the processing is completed; if the matching rate of the branch in the corresponding second node is not lower than the fourth threshold and the tree structure data loaded in the computing memory of the current second node exceeds the preset video memory threshold, then the tree structure data loaded in the computing memory of the current second node is deleted according to the matching rate and usage rate.

15. The data processing system according to any one of claims 1 to 11, characterized in that: Any first node and any second node are used: if the load of the current node is lower than the preset load threshold and the throughput of the current node is lower than the threshold throughput threshold, or if the load fluctuation of the current node is greater than the preset fluctuation threshold, then when the current node is the first node, the current node is changed to the second node; when the current node is the second node, the current node is changed to the first node.

16. The data processing system according to any one of claims 1 to 11, characterized in that: The control node is used to: if the matching rate of the global tree structure in at least two second nodes is lower than a preset fifth threshold, merge the global tree structures in the at least two second nodes into the same second node in the at least two second nodes, and determine that there is no second node of the global tree structure in the at least two second nodes, and change the second node to the first node; wherein the global tree structure obtained by the merger is not greater than a preset sixth threshold.

17. The data processing system according to any one of claims 1 to 11, characterized in that: Any first node and any second node are used to: if the request waiting amount in the current node is higher than the preset waiting task amount, determine whether to change the node type of the current node based on the number of highest priority requests among all waiting requests in the current node and the throughput of the current node.

18. A data processing method, characterized in that: include: A control node in the data processing system obtains a plurality of model inference requests and distributes the plurality of model inference requests to M first nodes; The data processing system includes: the control node, and M first nodes and N second nodes deployed in a set ratio; Any first node in the data processing system performs character encoding on a received model inference request to obtain an encoding result set; matches the encoding result set with N pre-stored partial tree structures to obtain N matching results; and sends the N matching results to the control node; the N partial tree structures correspond one-to-one to the N second nodes; The control node determines a target node among the N second nodes based on the received matching results and the remaining space of N computing memories among the N second nodes, so that the target node obtains the encoding result set and its corresponding key-value data set, decodes the encoding result set and the key-value data set to obtain the corresponding model inference result, and constructs a leaf node based on the encoding result set and the key-value data set, and fills the leaf node into the global tree structure in itself; a character feature in the encoding result set corresponds to a key-value data in the key-value data set.

19. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to claim 18.

20. A non-volatile storage medium, characterized in that: Used for storing a computer program, wherein the computer program implements the method according to claim 18 when executed by a processor.

21. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to claim 18 is implemented.

Images