Storage method, device, storage medium, and program product

Abstract

The present application provides a storage method, a device, a storage medium, and a program product. The method comprises: determining a plurality of virtual groups formed by dividing a first memory and a plurality of data blocks corresponding to each of the virtual groups; for a target key-value vector to be stored for a target request processed by a model, find a first data block most recently allocated to the target request, wherein the first data block is located in a first virtual group; when the first data block meets a storage requirement, storing the target key-value vector into the first data block; when the first data block does not meet the storage requirement and a second data block adjacent to the first data block is idle, allocating the second data block to the target request and storing the target key-value vector; and when the second data block has been allocated, searching for a second virtual group in which the first of the data blocks is idle, allocating the first of the data blocks of the second virtual group to the target request, and storing the target key-value vector.

Description

[0001] Storage method, device, storage medium and program product technical field

[0002]

[0001] The present disclosure relates to the field of computer technology, and more particularly to a storage method, device, storage medium, and program product.

[0003] KV cache (key-value cache) is an important technology currently used to optimize model reasoning performance. Model reasoning refers to reasoning and outputting response information based on given input information. KV is a vector calculated from the input information during the model reasoning process and used for reasoning operations. Taking the input information as an example, model reasoning performs reasoning operations based on the KV of each token (the smallest unit that the model can process, such as a word or character) in the input text to obtain and output the response text. In practical applications, model reasoning requires multiple rounds of reasoning operations, each round being a time step. Typically, the model outputs only one token during reasoning at each time step. The KV of the token output at the current time step is concatenated with the KVs of the previous output tokens to infer the token for the next time step until the reasoning ends.

[0004] In order to reduce the amount of calculation and improve the efficiency of reasoning, the KV cache stores the KV of the current time step, and the next time step directly reads the stored result without recalculating. Since the memory occupied by KV storage is large, in order to reduce memory waste, the memory is usually divided into multiple data blocks, each of which can store the KV of a certain number of tokens. Since the data blocks do not need to be continuous in the memory space, memory waste can be greatly reduced.

[0005]

[0004] In practical applications, the model usually performs task reasoning for multiple pending requests simultaneously. Different data blocks are allocated for key-value storage for different pending requests. In traditional solutions, data blocks are allocated to multiple pending requests in sequence. For example, the first data block is allocated to the first pending request, and the second data block adjacent to the first data block is allocated to the second pending request. However, this implementation method results in low memory access efficiency and poor storage effect.

[0006]

[0005] The embodiments of the present disclosure provide a storage method, device, storage medium, and program product to solve the problem in related technologies that key-value vectors are stored in a cross-discrete manner in physical space, which affects memory access efficiency.

[0007]

[0006] In a first aspect, the present disclosure provides a storage method, comprising: determining a plurality of virtual groups divided into a first memory and a plurality of data blocks corresponding to the plurality of virtual groups; searching for a first data block most recently allocated to a target key-value vector to be stored for a target request processed by a model; the first data block being located in the first virtual group; storing the target key-value vector in the first data block if the first data block meets the storage requirement; allocating the second data block to the target request and storing the target key-value vector in the second data block if the first data block does not meet the storage requirement and a second data block adjacent to the first data block is idle; and searching for a second virtual group whose first data block is idle if the second data block has been allocated, allocating the first data block of the second virtual group to the target request and storing the target key-value vector in the first data block of the second virtual group.

[0008]

[0007] In a second aspect, the present disclosure provides a computing device comprising multiple memories and multiple processors; the multiple memories provide a first memory and a second memory; the first memory and the second memory are used to store key-value vectors; wherein the multiple memories store one or more computer program instructions for the first processor to call and execute, so as to implement the storage method described in the first aspect.

[0009]

[0008] In a third aspect, the present disclosure provides a computer-readable storage medium storing a computer program, wherein the computer program is executed by a computer to implement the storage method as described in the first aspect.

[0010]

[0009] In a fourth aspect, the present disclosure provides a computer program product storing a computer program, wherein the computer program is executed by a computer to implement the storage method as described in the first aspect.

[0011] In an embodiment of the present disclosure, multiple virtual groups divided into a first memory and multiple data blocks corresponding to the multiple virtual groups can be determined. For a target key-value vector to be stored for a target request processed by a model, the first data block most recently allocated to the target request can be searched, where the first data block is located in the first virtual group. If the first data block meets the storage requirement, the target key-value vector can be stored in the first data block. If the first data block does not meet the storage requirement and a second data block adjacent to the first data block is free, the second data block can be allocated to the target request, and the target key-value vector can be stored in the second data block. If the first data block does not meet the storage requirement and the second data block has been allocated, a second virtual group with a free first data block can be searched, the first data block of the second virtual group can be allocated to the target request, and the target key-value vector can be stored in the first data block of the second virtual group. By dividing the system into multiple virtual groups, if the first data block in the first virtual group most recently allocated to a target request does not meet the storage requirements and the adjacent second data block has already been allocated to another target request, a second virtual group with an empty first data block is found and allocated to the target request to store the target key-value vector. This ensures that key-value vectors of the same request are stored in the same virtual group as much as possible, reduces the cross-storage of key-value vectors of different requests in physical space, thereby improving the continuity of key-value vector storage and further enhancing memory access efficiency.

[0012]

[0011] These and other aspects of the present disclosure will become more readily apparent in the following description of the embodiments.

[0013]

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings described below are some embodiments of the present disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0014] FIG1 shows a schematic structural diagram of an embodiment of data block division provided by the present disclosure;

[0015] FIG2 shows a schematic diagram of a system architecture according to an embodiment of the present invention;

[0016] FIG3 shows a flow chart of an embodiment of a storage method provided by the present disclosure;

[0017] FIG4 shows a schematic structural diagram of an embodiment of virtual grouping and data block division provided by the present disclosure;

[0018] FIG5 shows a flow chart of another embodiment of a storage method provided by the present disclosure;

[0019] FIG6 shows a schematic structural diagram of an embodiment of a storage device provided by the present disclosure;

[0020] FIG. 7 shows a schematic diagram of a structure of an embodiment of a computing device provided by the present disclosure.

[0021]

[0020] In order to enable people skilled in the art to better understand the solution of the present disclosure, the technical solution in the embodiment of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiment of the present disclosure.

[0022] Some of the processes described in the specification and claims of this disclosure and the accompanying drawings include multiple operations that appear in a specific order. However, it should be understood that these operations may not be executed in the order in which they appear herein or may be executed in parallel. Operation sequence numbers, such as 101 and 102, are merely used to distinguish between different operations and do not represent any specific order of execution. Furthermore, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that terms such as "first" and "second" herein are used to distinguish between different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0023] The technical solution disclosed herein is applicable to the field of computer technology. In conventional solutions, for multiple different pending requests for reasoning tasks executed by a model, data blocks are typically allocated sequentially for each pending request. Referring to the schematic diagram of data block partitioning and key-value (kv) storage shown in FIG1 , for example, data blocks 1 and 2 are allocated to request 1 to store the key-value vector of request 1, and data block 3 is allocated to request 2 to store the key-value vector of request 2. When the storage capacity of data block 2 reaches a threshold, data block 4 is allocated to request 1 to continue storing the key-value vector of request 1. When the storage capacity of data block 3 reaches a threshold, data block 5 is allocated to request 2 to continue storing the key-value vector of request 2. This process is repeated until reasoning for the multiple pending requests is completed.

[0024] As can be seen, in the above implementation, the KVs corresponding to different pending requests are stored in a discrete, cross-wise manner in physical space, and the data blocks corresponding to the KV stored for the same pending request are discontinuous. Therefore, when reasoning about a pending request and reading the KV stored for that request, multiple read operations are required for multiple discontinuous data blocks, resulting in low memory access efficiency.

[0025]

[0024] In order to solve the above technical problems, the inventors have proposed the technical solution of the present disclosure. In an embodiment of the present disclosure, multiple virtual groups divided into a first memory and multiple data blocks corresponding to the multiple virtual groups are determined; for a target key-value vector to be stored for a target request processed by a model, the first data block most recently allocated to the target request is searched; the first data block is located in the first virtual group; if the first data block meets the storage requirement, the target key-value vector is stored in the first data block; if the first data block does not meet the storage requirement and a second data block adjacent to the first data block is idle, the second data block is allocated to the target request, and the target key-value vector is stored in the second data block; if the second data block has been allocated, a second virtual group with an idle first data block is searched, the first data block of the second virtual group is allocated to the target request, and the target key-value vector is stored in the first data block of the second virtual group.

[0026]

[0025] In the embodiment of the present disclosure, by dividing into multiple virtual groups, when the first data block in the first virtual group most recently allocated to the target request does not meet the storage requirements and the second data block adjacent to the first data block has also been allocated to other target requests, a second virtual group with an idle first data block is found and allocated to the target request to store the target key-value vector, thereby ensuring that the key-value vectors of the same request are stored as much as possible in the same virtual group, reducing the cross-storage of key-value vectors of different requests in physical space, thereby improving the continuity of key-value vector storage and further improving memory access efficiency.

[0027] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings. It is apparent that the described embodiments are only a portion of the embodiments of the present disclosure, and are not intended to be exhaustive. All other embodiments derived by those skilled in the art based on the embodiments of the present disclosure without creative effort are intended to fall within the scope of protection of the present disclosure.

[0028]

[0027] The technical solution of the embodiment of the present disclosure can be applied to the system architecture shown in FIG2 , which may include at least one client 201 and a server 202. Each client 201 establishes a connection with the server 202 via a network. Each client 201 can send a request to the server 202 via the network, so that the server 202 performs inference based on the request using a model and feeds back the inference result to the client 201.

[0028] The client 201 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Languages, version 5) application, or a light application (also known as a mini-program, a lightweight application), or a cloud application. The client can be deployed in an electronic device and needs to rely on the device to run or on certain apps in the device to run. The electronic device can, for example, have a display screen and support information browsing, and can be a personal mobile terminal such as a mobile phone, a tablet computer, or a personal computer.

[0029]

[0029] The server 202 may include servers that provide various services, such as servers that process requests sent by the client 201, servers that train models, and the like. It should be noted that the server may be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server may also be a server in a distributed system, or a server integrated with a blockchain. The server may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0030]

[0030] It should be noted that the embodiments of the present disclosure may involve the use of user data. In actual applications, user-specific personal data can be used in the solutions described herein within the scope permitted by applicable laws and regulations of the country where the user is located (for example, with the user's explicit consent, effective notification to the user, etc.).

[0031]

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0032]

[0032] It should be noted that the technical solution of the embodiment of the present disclosure is applicable to a network virtual environment. The user described generally refers to a "virtual user". A real user can register a user account in the server through a registration method to obtain a user identity in the network environment.

[0033]

[0033] FIG3 is a flow chart of an embodiment of a storage method provided by an embodiment of the present disclosure. The technical solution of this embodiment can be executed by the server. The method may include the following steps.

[0034]

[0034] 301: Determine multiple virtual groups divided from the first memory and multiple data blocks corresponding to the multiple virtual groups.

[0035] In practical applications, the server can be deployed in a computing device, which can include multiple processors, such as a GPU (graphics processing unit), a CPU (central processing unit), and the like. To improve computing efficiency, the technical solution of this embodiment can be executed by a GPU, and the first memory can be the GPU's video memory.

[0036]

[0036] The server may pre-divide the first memory into a plurality of virtual groups and a plurality of data blocks corresponding to the plurality of virtual groups. As shown in FIG4 , a schematic diagram of the division of virtual groups and data blocks is shown.

[0037]

[0037] The size and number of the virtual packets, as well as the size and number of the data blocks, can be set according to actual needs, or there may be other determination methods, which will be described in subsequent embodiments.

[0038]

[0038] 302: For the target key value vector to be stored for the target request processed by the model, search for the first data block most recently allocated to the target request, where the first data block is located in the first virtual group.

[0039]

[0039] Optionally, the models involved in the embodiments of the present disclosure may be implemented as a large model (also called a foundation model). A large model refers to a machine learning model with a large number of parameters and a complex structure. It is capable of processing massive amounts of data and completing various complex tasks, such as natural language processing, computer vision, and speech recognition. It is an AI (artificial intelligence) model. The large model can be implemented using a large language model (LLM) or a multimodal large model (MLM). This disclosure does not limit this.

[0040]

[0040] In practical applications, a model can process one or more requests. For any request, a key-value vector can be stored according to the solution of the embodiments of the present disclosure. The target request refers to any request, and the target key-value vector can refer to any key-value vector. The request can include initial input data, i.e., a prompt. Prompt is a form of input that can include a specific task or instruction that the model is expected to perform. It is used to prompt or guide the model to produce expected output, indicating what action the model should take or what output it should generate when performing a specific task.

[0041] The target key-value vector can be a key-value vector calculated from any text unit obtained by segmenting the prompt in the pre-filling phase. Alternatively, it can be a key-value vector calculated from any text unit output at each time step in the decoding phase. The model inference process can be divided into two main phases: the pre-filling phase (i.e., the phase in which the first text unit is generated based on the prompt) and the decoding phase (the phase in which the second and subsequent text units are generated). In the embodiments of the present disclosure, a text unit can be, for example, a token (the smallest unit that the model can process, such as a word or character).

[0042] For the target key-value vector to be processed for the target request, a data block most recently allocated to the target request can be searched. For ease of description, this data block can be referred to as the first data block, and the first data block is located in the first virtual group. The first data block most recently allocated to the target request can refer to the first data block allocated to the target request in the previous time step, i.e., the first data block storing the key-value vector to be stored for the target request in the previous time step.

[0043]

[0043] It can be understood that, when there is a first data block that was most recently allocated to the target request, that is, when the first data block was allocated to the target request in the previous time step, the above-mentioned operation of searching for the first data block can be performed, and when there is no first data block that was most recently allocated to the target request, such as when the current time step is the initial time step, the operation of allocating the first data block to the target request can be performed.

[0044]

[0044] Therefore, optionally, for the target key value vector to be stored for the target request processed by the model, searching for the first data block most recently allocated to the target request may include: for the target key value vector to be stored for the target request, when the target key value vector is generated in the pre-filling stage based on the text units included in the target request, allocating data blocks in at least one virtual group to the target request, and storing the target key value vector in the data blocks in the at least one virtual group in sequence; when the target key value vector is generated in the decoding stage based on the text units produced at any time step, searching for the first data block most recently allocated to the target request.

[0045] In the pre-filling stage, the target key vector is generated based on the text units included in the target request. That is, the current time step is the initial time step, and no data blocks have been allocated to the target request before this. In this case, the data blocks in at least one virtual group can be allocated to the target request, and the target key vector can be sequentially stored in the data blocks in the at least one virtual group. There are various implementation methods.

[0046]

[0046] As an optional implementation, a data block in at least one virtual group can be randomly selected from the first memory and allocated to the target request.

[0047] As another optional implementation, it may be prioritized to determine whether there are idle virtual groups in the first memory. An idle virtual group may indicate that the data blocks in the virtual group are not allocated to any request and do not store any key vectors for any request. If the determination result is yes, at least one data block in the idle virtual group, or at least one data block in the idle virtual group, may be allocated to the target request, thereby sequentially storing the target key vectors in the allocated data blocks. If the determination result is no, at least one data block in the virtual group with an idle data block, or at least one data block in the virtual group with an idle data block, may be allocated to the target request.

[0048] By preferentially selecting data blocks in an idle virtual group to allocate to the target request at the initial time step, cross-storage of key-value vectors of other requests in the data blocks of the same virtual group is greatly reduced, thereby improving the continuity of key-value vector storage and further improving memory access efficiency.

[0049] In the decoding phase after the pre-filling phase, the target key vector is generated based on the text units produced at any time step, that is, the current time step is any time step after the initial time step, and the data block has been allocated to the target request in the previous time step. At this time, the first data block most recently allocated to the target request can be searched.

[0050]

[0050] 303: Detect whether the first data block meets the storage requirement. If the first data block meets the storage requirement, perform the operation of step 304; otherwise, perform the operation of step 305.

[0051]

[0051] The storage requirement may include, for example, that there is storage space for the first data block, or that the first data block does not reach a storage capacity threshold, etc., and may be set according to actual needs.

[0052] 304: Store the target key-value vector into the first data block.

[0052]

[0053] When the first data block meets the storage requirement, the target key-value vector can be stored in the first data block, thereby achieving continuous storage of the key-value vector in the data block.

[0053]

[0054] 305: Detect whether a second data block adjacent to the first data block is free. If the second data block is free, execute the operation of step 306; if the second data block is allocated, execute the operation of step 307.

[0054]

[0055] When the first data block does not meet the storage requirement, the target key-value vector cannot be stored in the first data block. At this time, the next data block adjacent to the first data block can be detected, which can be called the second data block for ease of description, to see whether it is free.

[0055]

[0056] The free data block may indicate that the data block is not allocated to any request and does not store any key-value vector of any request.

[0056]

[0057] 306: Allocate a second data block to the target request, and store the target key-value vector in the second data block.

[0057]

[0058] When the second data block is idle, the second data block may be allocated to the target request, and the target key-value vector may be stored in the second data block.

[0058]

[0059] 307: Find a second virtual group with an idle first data block, allocate the first data block of the second virtual group to the target request, and store the target key-value vector in the first data block of the second virtual group.

[0059]

[0060] When the second data block has been allocated, it indicates that the second data block has been allocated to another request. At this time, in order to reduce the cross-storage of key-value vectors of different requests in space, a second virtual group with an idle first data block can be found, and the first data block in the second virtual group can be allocated to the target request, and the target key-value vector can be stored in the first data block of the second virtual group.

[0060]

[0061] In practical applications, after the target key-value vector to be stored in the target request is stored in the data block, the storage information of the target key-value vector may also be recorded in the block table.

[0061]

[0062] The stored information of the target key-value vector may include identification information of the stored data block, identification information of the virtual group in which the data block resides, and identification information of the corresponding target request. The record information in the block table may include identification information of the virtual group corresponding to each target request, identification information of the data block in the virtual group, and the number of text units stored in the data block.

[0062]

[0063] Optionally, the first data block most recently allocated by the target request may be determined according to the record information in the block table, and it may be determined whether the first data block meets the storage requirement and whether the second data block is idle, thereby improving the determination efficiency.

[0063]

[0064] In this embodiment, multiple virtual groups divided into the first memory and multiple data blocks corresponding to the multiple virtual groups can be determined. For a target key-value vector to be stored for a target request processed by the model, the first data block most recently allocated to the target request can be searched, where the first data block is located in the first virtual group. If the first data block meets the storage requirement, the target key-value vector can be stored in the first data block. If the first data block does not meet the storage requirement and a second data block adjacent to the first data block is free, the second data block can be allocated to the target request, and the target key-value vector can be stored in the second data block. If the first data block does not meet the storage requirement and the second data block is already allocated, a second virtual group with a free first data block can be searched, the first data block of the second virtual group can be allocated to the target request, and the target key-value vector can be stored in the first data block of the second virtual group. By dividing the target request into multiple virtual groups, if the first data block in the first virtual group most recently allocated to the target request does not meet the storage requirements and the adjacent second data block has already been allocated to another target request, the target request can store the target key vector by finding a second virtual group with an empty first data block. This ensures that key vectors of the same request are stored in the same virtual group as much as possible, reduces the cross-storage of key vectors of different requests in physical space, thereby improving the continuity of key vector storage and further enhancing memory access efficiency.

[0064]

[0065] The virtual grouping and data block pre-division process is described below.

[0065]

[0066] In some embodiments, the above method may include: using the virtual group size, the number of virtual groups, the data block size, the number of data blocks, and the maximum request processing number as multiple hyperparameters corresponding to the model; finding candidate parameter value combinations for the multiple hyperparameters; using test samples to perform a model evaluation on the model set according to the candidate parameter value combination, and if the model evaluation result meets the performance requirements, using the candidate parameter value combination as the target parameter value combination; and dividing the first memory into multiple virtual groups and multiple data blocks corresponding to the multiple virtual groups according to the target virtual group size, target number of virtual groups, target data block size, and target number of data blocks determined in the target parameter value combination.

[0066]

[0067] The maximum number of request processing may refer to the number of target requests that the model can process simultaneously. When the number of received requests exceeds the maximum number of request processing, the excess requests may be placed in a request pool for processing.

[0067]

[0068] Specifically, when searching for candidate parameter value combinations for multiple hyperparameters, the maximum number of text units that the model can process can be preferentially pre-set based on the actual application scenario. To improve the rationality of setting the maximum number of text units, the utilization rate of the first memory during full load operation can be set based on the actual application scenario. For example, it can be set to 95%. The available first memory size can be calculated by multiplying the size of the first memory by the utilization rate. The maximum number of text units does not exceed the available first memory size. For example, the number of text units can be set to 1024, etc.

[0068]

[0069] Furthermore, the virtual group size is multiplied by the number of virtual groups, and the result obtained by multiplying the data block size by the number of data blocks can be consistent with the maximum number of text units processed. In addition, multiple historical requests can be obtained, and the maximum number of request processing is multiplied by the average number of historical requests. The result obtained by multiplying the maximum number of request processing is also consistent with the maximum number of request processing, thereby obtaining multiple candidate parameter value combinations for hyperparameters.

[0069]

[0070] Afterwards, the test sample is used to perform a model evaluation on the model set according to the candidate parameter value combination. If the model evaluation result meets the performance requirements, the candidate parameter value combination is used as the target parameter value combination, and multiple virtual groups and data blocks are divided accordingly. The size of the data block can be set to 16, for example, indicating that the key-value vectors corresponding to 16 text units can be stored. The size of the virtual group can be set to 8, for example, indicating that 8 data blocks can be included, and so on.

[0070]

[0071] By dividing virtual groups and achieving a reasonable setting of data block size, we can reduce storage waste within the data blocks and further reduce memory waste. On this basis, we can reduce the cross-storage of key-value vectors of different requests in physical space, improve the continuity of key-value vector storage, and improve memory access efficiency.

[0071]

[0072] Optionally, using the test sample to evaluate the model set according to the candidate parameter value combination may include: powdering the test sample into the model set according to the candidate parameter value combination to obtain a prediction result; and determining the model evaluation result based on the difference information between the prediction result and the expected result.

[0072]

[0073] By performing a hyperparameter search to determine the size and number of virtual groups and data blocks, the first memory is divided, thereby improving the rationality of the virtual group and data block settings and further reducing the waste of first memory resources.

[0073]

[0074] Optionally, the above virtual grouping and data block division information may be recorded in a block table.

[0074]

[0075] The following will continue to explain the allocation phase, the allocation of data blocks and the storage process of key-value vectors.

[0075]

[0076] It is understandable that in actual applications, when searching for a second virtual group with an idle first data block, if the search result for the second virtual group is not empty, that is, if a second virtual group with an idle first data block exists in the first memory, the first data block of the second virtual group can be allocated to the target request, and the target key-value vector can be stored in the first data block of the second virtual group. There may be multiple second virtual groups with idle first data blocks in the first memory, and any second virtual group with an idle first data block can be selected for allocation without restriction.

[0076]

[0077] When the search result of the second virtual group is empty, it indicates that there is no second virtual group with a free first data block in the first memory. At this time, there may be multiple implementation methods.

[0077]

[0078] As an optional implementation method, the historical key-value vector stored in the data block corresponding to the target request can be transferred to the second memory, and the data block corresponding to the target request can be cleared. Then, the first data block in the cleared third virtual group is allocated to the target request, and the target key-value vector is stored in the first data block in the third virtual group.

[0079] The second memory and the first memory correspond to different processors. Optionally, the second memory may be a CPU memory. When the first memory does not contain a second virtual group with a free first data block, all historical key-value vectors of the target request stored in the first memory before the current time step can be transferred to the second memory, and the corresponding data blocks can be cleared. Subsequently, if the cleared data blocks include a third virtual group with a free first data block, the first data block in the third virtual group can be assigned to the target request, and the target key-value vector can be stored in the first data block in the third virtual group, thereby improving the continuity of key-value vector storage.

[0078]

[0080] When there is no second virtual group with a free first data block in the first memory, the stored historical key-value vector of the target request is transferred to the second memory for storage, and the corresponding data block is cleared for continued storage of the key-value vector in the current time step and subsequent time steps. This ensures the continuity of key-value vector storage while achieving the storage of the target key-value vector.

[0079]

[0081] Optionally, the record information of the target request in the block table may be updated according to the storage information of the target key-value vector.

[0080]

[0082] As another optional implementation of the second virtual grouping when no free first data block exists in the first memory, a free data block may be randomly selected and allocated to the target request, and the target key-value vector may be stored in the free data block.

[0081]

[0083] By randomly selecting free data blocks, the target key-value vector can be stored in the first memory without having to transfer the stored historical key-value vectors between different memories, thus avoiding resource consumption and improving memory access efficiency.

[0082]

[0084] Of course, when there are free data blocks in the first memory, a free data block is randomly selected and allocated to the target request. Optionally, whether there is a second virtual group with free first data blocks and whether there are free data blocks can be determined based on the record information in the block table.

[0083]

[0085] Alternatively, whether there are free data blocks may be determined in combination with usage of data blocks in the first memory. In some embodiments, if the search result of the second virtual group is empty, randomly selecting a free data block to allocate to the target request and storing the target key vector in the free data block may include: if the search result of the second virtual group is empty and the data blocks in the first memory meet usage conditions, randomly selecting a free data block to allocate to the target request and storing the target key vector in the free data block.

[0084]

[0086] The usage condition may, for example, include that the usage rate of data blocks in the first memory is lower than a usage rate threshold. The usage rate threshold may be, for example, 99% and may be set based on actual needs. The usage rate may be determined based on the ratio of the number of allocated data blocks to the number of partitioned data blocks in the first memory. Alternatively, the usage condition may also include that the remaining rate of data blocks in the first memory is higher than a remaining rate threshold. The remaining rate threshold may be, for example, 1%. The remaining rate may be determined based on the difference between a value of 1 and the ratio of the number of allocated data blocks to the number of partitioned data blocks in the first memory. Other usage conditions may also be set based on actual needs and are not limited in this disclosure.

[0085]

[0087] When there is no free second virtual group of the first data block in the first memory and the data block in the first memory meets the usage condition, it indicates that there is a free data block in the first memory. At this time, a free data block can be randomly selected and allocated to the target request, and the target key-value vector is stored in the free data block.

[0086]

[0088] When there is no free second virtual group of the first data block in the first memory and the data in the first memory does not meet the usage condition, it indicates that there is no free data block in the first memory and there is no space in the first memory to store the target key-value vector. In this case, multiple implementation methods can be used.

[0087]

[0089] As an optional implementation method, the historical key-value vector stored in the data block corresponding to the target request can be transferred to the second memory, and the data block corresponding to the target request can be cleared. Then, the first data block in the cleared third virtual group is allocated to the target request, and the target key-value vector is stored in the first data block in the third virtual group.

[0088]

[0090] If there is no space in the first memory to store the target key-value vector, all historical key-value vectors for the target request stored in the first memory before the current time step can be transferred to the second memory, and the corresponding data blocks can be cleared. Subsequently, a randomly selected cleared data block can be assigned to the target request, and the target key-value vector can be stored in this data block.

[0089]

[0091] Alternatively, a determination may be made prior to determining whether the cleared data blocks include a third virtual group whose first data block is free. If so, the first data block in the third virtual group may be allocated to the target request, and the target key-value vector may be stored in the first data block in the third virtual group to improve the continuity of key-value vector storage. If not, a randomly selected cleared data block may be allocated to the target request, and the target key-value vector may be stored in the data block.

[0090]

[0092] The storage of key-value vectors is achieved by transferring the stored historical key-value vectors of the target request to the second memory for storage when there is no storage space for the target key-value vector in the first memory, and clearing the corresponding data block for continuing to store the key-value vector in the current time step and subsequent time steps.

[0091]

[0093] As another optional implementation method when there is no space in the first memory to store the target key-value vector, the target key-value vector may not be stored. The target key-value vector is used to be recalculated according to its corresponding text unit when reasoning is performed at the next time step.

[0092]

[0094] By not storing the target key-value vector, there is no need to transfer the stored historical key-value vectors between different memories, which avoids resource consumption and improves memory access efficiency.

[0093]

[0095] In this case, target key-value vectors not stored in the current time step cannot be directly retrieved from memory at the next time step. They must be recalculated based on their corresponding text units during inference at the next time step to implement model inference. Considering the computational resource consumption and impact on model inference efficiency caused by recalculating key-value vectors, infinite recalculation can be avoided.

[0094]

[0096] Therefore, in some embodiments, when the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition, not storing the target key-value vector may include: when the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition, if the recalculated number of text units is less than a predetermined value, not storing the target key-value vector.

[0095]

[0097] Furthermore, the method may include: if the number of recalculated text units reaches a predetermined value, transferring the historical key-value vector stored in the data block corresponding to the target request to the second memory, and clearing the data block corresponding to the target request. Thereafter, allocating the first data block in the cleared third virtual group to the target request, and storing the target key-value vector in the first data block in the third virtual group.

[0096]

[0098] The setting of this predetermined value can be used to balance the resource consumption caused by recalculating the key-value vector and the impact on model inference efficiency, and the resource consumption caused by transferring historical key-value vectors between different memories and the impact on memory access efficiency. For example, it can be set to 4, 5, etc., and can be determined based on actual application scenarios.

[0097]

[0099] By setting a predetermined value, if there is no second virtual group with a free first data block in the first memory, the data block does not meet the usage conditions, and the number of recalculated text units is less than the predetermined value, the target key-value vector is not stored and recalculated based on the corresponding text unit during inference at the next time step. If the number of recalculated text units reaches the predetermined value, the historical key-value vector stored in the data block corresponding to the target request is transferred to the second memory, the data block corresponding to the target request is cleared, the first data block in the cleared third virtual group is assigned to the target request, and the target key-value vector is stored in the first data block in the third virtual group. This method can achieve a balance between the resource consumption caused by recalculating key-value vectors and the impact on model inference efficiency, and the resource consumption caused by transferring historical key-value vectors between different memories and the impact on memory access efficiency, thereby achieving optimal key-value vector storage, calculation, memory access, and model inference performance.

[0098]

[0100] One or more of the above embodiments provide a storage method for key-value vectors. The following describes a process of performing model inference based on the stored key-value vectors.

[0099]

[0101] In some embodiments, the above method may further include: calculating a target key-value vector corresponding to a text unit generated by a target request at a previous time step; reading a historical key-value vector corresponding to the target request from a first memory and / or a second memory; concatenating the target key-value vector with the historical key-value vector to generate an input vector for the current time step; and using a model to generate a text unit corresponding to the current time step based on the input vector.

[0100]

[0102] Specifically, at the current time step, the target key-value vector corresponding to the text unit generated at the previous time step can be calculated, and the historical key-value vector corresponding to the target request can be read from the first memory and / or the second memory that stores the historical key-value vector. The read historical key-value vector is concatenated with the target key-value vector calculated at the current time step to obtain the input vector of the model at the current time step. The input vector is then input into the model to infer the text unit corresponding to the current time step.

[0101]

[0103] Optionally, reading the historical key-value vector corresponding to the target request from the first memory and / or the second memory may include: reading the historical key-value vector corresponding to the target request from the first memory and / or the second memory according to the record message of the target request in the block table.

[0102]

[0104] The above process illustrates a model inference process that does not require recalculation of key-value vectors. However, as discussed in the previous embodiments, recalculation of key-value vectors may be necessary. Therefore, in some embodiments, the method may further include: determining whether at least one text unit needs to be recalculated; and if so, calculating at least one key-value vector corresponding to the at least one text unit.

[0103]

[0105] At this time, concatenating the target key-value vector and the historical key-value vector to generate the input vector of the current time step may include: concatenating the target key-value vector, at least one key-value vector, and the historical key-value vector to generate the input vector of the current time step.

[0104]

[0106] Specifically, at the current time step, if it is determined that there is at least one text unit to be recalculated, at least one key-value vector corresponding to the at least one text unit and a target key-value vector corresponding to the text unit generated at the previous time step can be calculated, and the historical key-value vector corresponding to the target request can be read from the first memory and / or the second memory storing the historical key-value vector. The read historical key-value vector is concatenated with at least one key-value vector corresponding to the at least one text unit recalculated at the current time step and the target key-value vector corresponding to the text unit generated at the previous time step and calculated at the current time step to obtain an input vector of the model at the current time step. The input vector is input into the model to infer the text unit corresponding to the current time step.

[0105]

[0107] For ease of understanding, in an actual application scenario, FIG5 shows a schematic diagram of a process for allocating data blocks and storing key-value vectors, which may include the following steps.

[0106]

[0108] 500: Pre-divide a first memory into a plurality of virtual groups and a plurality of data blocks corresponding to the plurality of virtual groups, and record the data blocks in a block table.

[0107]

[0109] 501: At an initial time step, allocating at least one idle virtual grouped data block in a first memory to the target request.

[0108]

[0110] 502: At any time step after the initial time step, the following steps are executed in a loop until the target request processing is completed.

[0111] 5020: Search for the first data block most recently allocated by the target request, where the first data block is located in the first virtual group.

[0109]

[0112] 5021: Detect whether the first data block meets the storage requirement. If the first data block meets the storage requirement, execute the operation of step 5022; otherwise, execute the operation of step 5023.

[0110]

[0113] 5022: Store the target key-value vector into the first data block.

[0111]

[0114] 5023: Detect whether a second data block adjacent to the first data block is free. If the second data block is free, execute the operation of step 5024; if the second data block is allocated, execute the operation of step 5025.

[0112]

[0115] 5024: Allocate a second data block to the target request, and store the target key-value vector in the second data block.

[0113]

[0116] 5025: Check whether there is a second virtual group with an idle first data block. If so, execute the operation of step 5026; otherwise, execute the operation of step 5027.

[0114]

[0117] 5026: Allocate the first data block of the second virtual group to the target request, and store the target key-value vector in the first data block of the second virtual group.

[0115]

[0118] 5027: Check whether the data block meets the usage conditions. If so, execute the operation of step 5028; otherwise, execute the operation of step 5029.

[0116]

[0119] 5028: Randomly select a free data block and allocate it to the target request, and store the target key-value vector in the free data block.

[0117]

[0120] 5029: Determine whether the recalculated number of text units is less than a predetermined value. If so, execute step 5030; otherwise, execute step 5031.

[0118]

[0121] 5030: Target key-value vector not stored.

[0119]

[0122] 5031: Transfer the historical key-value vector stored in the data block corresponding to the target request to the second memory, clear the corresponding data block, assign the first data block in the cleared third virtual group to the target request, and store the target key-value vector in the first data block in the third virtual group.

[0120]

[0123] In this embodiment, the specific implementation of steps 500-502, and 5020-5031 has been described in the embodiment of FIG3 and will not be repeated here.

[0121]

[0124] In this embodiment, by dividing the data block into multiple virtual groups, a reasonable setting of the data block size is achieved, which can reduce storage waste within the data block and further reduce memory waste. On this basis, if the first data block in the first virtual group most recently allocated to the target request does not meet the storage requirements and the adjacent second data block has already been allocated to another target request, a second virtual group with an empty first data block is found and allocated to the target request to store the target key-value vector. This ensures that key-value vectors for the same request are stored as much as possible in the same virtual group, reducing the cross-storage of key-value vectors for different requests in physical space. This improves the continuity of key-value vector storage and, in turn, memory access efficiency. Furthermore, by setting a predetermined value, if there is no empty second virtual group with an empty first data block in the first memory, the data block does not meet the usage conditions, and the number of recalculated text units is less than the predetermined value, the target key-value vector is not stored and recalculated based on its corresponding text unit during inference at the next time step. If the number of recalculated text units reaches the predetermined value, the historical key-value vector stored in the data block corresponding to the target request is transferred to the second memory, and the data block corresponding to the target request is cleared. And allocating the first data block in the cleared third virtual group to the target request, and storing the target key-value vector in the first data block in the third virtual group, can achieve a balance between the resource consumption caused by recalculating the key-value vector and the impact on model inference efficiency, and the resource consumption caused by transferring historical key-value vectors between different memories and the impact on memory access efficiency, thereby achieving a better degree of key-value vector storage, calculation, memory access, and model inference effects.

[0122]

[0125] FIG6 is a schematic diagram of the structure of an embodiment of a storage device provided by the present disclosure. The device may include the following modules.

[0123]

[0126] The first determining module 601 is configured to determine a plurality of virtual groups divided into the first memory and a plurality of data blocks corresponding to the plurality of virtual groups.

[0124]

[0127] A search module 602 is configured to search for a first data block most recently allocated to a target key-value vector to be stored for a target request processed by the model; the first data block is located in a first virtual group.

[0125]

[0128] The first storage module 603 is configured to store the target key-value vector in the first data block when the first data block meets the storage requirement.

[0126]

[0129] The second storage module 604 is configured to allocate the second data block to the target request and store the target key-value vector in the second data block when the first data block does not meet the storage requirement and a second data block adjacent to the first data block is idle.

[0127]

[0130] The third storage module 605 is configured to, when the second data block has been allocated, search for a second virtual group whose first data block is free, allocate the first data block of the second virtual group to the target request, and store the target key-value vector in the first data block of the second virtual group.

[0128]

[0131] In some embodiments, the search module 602 may include: an allocation unit, for a target key-value vector to be stored for a target request, when the target key-value vector is generated in a pre-filling phase based on a text unit included in the target request, for allocating data blocks in at least one virtual group to the target request, and storing the target key-value vector in sequence in the data blocks in the at least one virtual group; a search unit, for searching for the first data block most recently allocated to the target request, when the target key-value vector is generated in a decoding phase based on a text unit produced at any time step.

[0129]

[0132] In some embodiments, the apparatus may further include: a fourth storage module, configured to randomly select an idle data block to allocate to the target request and store the target key-value vector in the idle data block when the search result of the second virtual group is empty.

[0130]

[0133] In some embodiments, the fourth storage module may be configured to randomly select an idle data block to allocate to the target request and store the target key-value vector in the idle data block when the search result of the second virtual group is empty and the data block in the first memory meets the usage condition.

[0131]

[0134] The device may also include: a second determination module, which is used to not store the target key-value vector when the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition; the target key-value vector is used to recalculate according to its corresponding text unit when reasoning is performed in the next time step.

[0132]

[0135] In some embodiments, the second determining module may be configured to not store the target key-value vector if the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition and the number of recalculated text units is less than a predetermined value.

[0133]

[0136] The device may also include: a transfer module, which is used to transfer the historical key-value vector stored in the data block corresponding to the target request to a second memory and clear the data block corresponding to the target request if the number of recalculated text units reaches the predetermined value; the second memory and the first memory correspond to different processors; and a fifth storage module, which is used to assign the first data block in the cleared third virtual group to the target request and store the target key-value vector in the first data block in the third virtual group.

[0134]

[0137] In some embodiments, the device may further include: a third determination module, configured to use the virtual group size, data block size, number of data blocks, and maximum request processing number as multiple hyperparameters corresponding to the model; a search module, configured to find candidate parameter value combinations of the multiple hyperparameters; an evaluation module, configured to use a test sample to perform a model evaluation on the model set according to the candidate parameter value combination, and if the model evaluation result meets the performance requirements, use the candidate parameter value combination as the target parameter value combination; a partitioning module, configured to partition the first memory into multiple virtual groups and multiple data blocks corresponding to the multiple virtual groups according to the target virtual group size, target data block size, and target number of data blocks determined in the target parameter value combination.

[0135]

[0138] In some embodiments, the evaluation module may be specifically configured to input a test sample into the model configured according to the candidate parameter value combination to obtain a prediction result; and determine a model evaluation result based on difference information between the prediction result and the expected result.

[0139] In some embodiments, the apparatus may further include: an updating module, configured to update record information of the target request in the block table according to storage information of the target key-value vector.

[0136]

[0140] In some embodiments, the device may further include: a first calculation module, used to calculate the target key-value vector corresponding to the text unit generated by the target request in the previous time step; a reading module, used to read the historical key-value vector corresponding to the target request from the first memory and / or the second memory; a first generation module, used to concatenate the target key-value vector with the historical key-value vector to generate an input vector for the current time step; and a second generation module, used to use the model to generate the text unit corresponding to the current time step based on the input vector.

[0137]

[0141] In some embodiments, the reading module may be specifically configured to read the history key-value vector corresponding to the target request from the first memory and / or the second memory according to the record message of the target request in the block table.

[0138]

[0142] In some embodiments, the device may further include: a judgment module, configured to judge whether there is at least one text unit to be recalculated; a second calculation module, configured to calculate at least one key-value vector corresponding to the at least one text unit when the result of the judgment module is yes; and a first generation module, specifically configured to concatenate the target key-value vector, the at least one key-value vector, and the historical key-value vector to generate a powder input vector for the current time step.

[0139]

[0143] FIG7 is a schematic diagram of the structure of an embodiment of a computing device provided by the present disclosure. The device may include at least multiple memories and multiple processors. The multiple memories provide a first memory 701 and a second memory 702, which may be used to store key-value vectors. The multiple processors may include a first processor 703 and a second processor 704. In practical applications, the first processor 703 may be, for example, a GPU, and the first memory may be the GPU's corresponding video memory. The second processor 704 may be, for example, a CPU, and the second memory may be the CPU's corresponding memory.

[0140]

[0144] The multiple memories store one or more computer program instructions for the first processor 703 to call and execute, so as to implement the storage method shown in FIG. 3 or FIG. 5 .

[0141]

[0145] Of course, the above computing device may also include other components, such as input / output interfaces, communication components, etc.

[0142]

[0146] The input / output interface provides an interface between the processing component and the peripheral interface module, which may be an output device, an input device, etc. The communication component is configured to facilitate wired or wireless communication between the computing device and other devices.

[0143]

[0147] It should be noted that the computing device may be a physical device or an elastic computing host provided by a cloud computing platform. It may be implemented as a distributed cluster consisting of multiple servers or terminal devices, or as a single server or a single terminal device.

[0144]

[0148] In one or more of the above embodiments, the processor may be implemented as one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components, and may be used to execute the above methods.

[0145]

[0149] Memory is configured to store various types of data to support operations in the terminal. Memory can be implemented by any type of volatile or nonvolatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0146]

[0150] The display may be an electroluminescent (EL) element, a liquid crystal display or a microdisplay having a similar structure, or a direct retinal display or a similar laser scanning display.

[0147]

[0151] Embodiments of the present disclosure also provide a computer-readable storage medium storing a computer program. When executed by a computer, the computer program can implement the storage method shown in FIG. 3 or FIG. 5 . The computer-readable medium can be included in the computing device described in the above embodiments, or it can exist independently and not be incorporated into the computing device.

[0148]

[0152] The computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof.

[0149]

[0153] An embodiment of the present disclosure further provides a computer program product, which includes a computer program carried on a computer-readable storage medium. When the computer program is executed by a computer, it can implement the storage method shown in FIG. 3 or FIG. 5 .

[0150]

[0154] In such an embodiment, the computer program may be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, various functions defined in the system of the present disclosure are performed.

[0151]

[0155] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152]

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. That is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the solutions in this embodiment. Persons of ordinary skill in the art can understand and implement the solutions without inventive effort.

[0153]

[0157] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or alternatively, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant art, can be embodied in the form of a software product. This software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or portions thereof.

[0154]

[0158] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present disclosure.

Claims

Claims 1. A storage method, comprising: Determine a plurality of virtual groups divided into the first memory and a plurality of data blocks corresponding to the plurality of virtual groups respectively; Searching, for a target key-value vector to be stored for a target request processed by the model, a first data block most recently allocated to the target request; the first data block being located in a first virtual group; and storing the target key-value vector in the first data block if the first data block meets storage requirements; When the first data block does not meet the storage requirement and a second data block adjacent to the first data block is free, allocating the second data block to the target request and storing the target key-value vector in the second data block; If the second data block has been allocated, find a second virtual group with a free first data block, allocate the first data block of the second virtual group to the target request, and store the target key-value vector in the first data block of the second virtual group.

2. The method according to claim 1, wherein: For the target key-value vector to be stored for the target request processed by the model, searching for the first data block most recently allocated to the target request includes: for the target key-value vector to be stored for the target request, when the target key-value vector is generated in the pre-filling stage based on the text unit included in the target request, allocating data blocks in at least one virtual group to the target request, and storing the target key-value vector in the data blocks in the at least one virtual group in sequence; when the target key-value vector is generated in the decoding stage based on the text unit produced at any time step, searching for the first data block most recently allocated to the target request.

3. The method according to claim 1, further comprising: When the search result of the second virtual group is empty, a free data block is randomly selected and allocated to the target request, and the target key-value vector is stored in the free data block.

4. The method according to claim 3, wherein: The randomly selecting an idle data block to allocate to the target request when the search result of the second virtual group is empty, and storing the target key-value vector in the idle data block includes: randomly selecting an idle data block to allocate to the target request when the search result of the second virtual group is empty and the data blocks in the first memory meet the usage condition, and storing the target key-value vector in the idle data block; the method further includes: When the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition, the target key-value vector is not stored; the target key-value vector is used to be recalculated according to its corresponding text unit when reasoning is performed at the next time step.

5. The method according to claim 4, wherein: The method of not storing the target key-value vector when the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition includes: when the search result of the second virtual group is empty and the data block in the first memory does not meet the usage condition, if the number of recalculated text units is less than a predetermined value, not storing the target key-value vector; the method also includes: if the number of recalculated text units reaches the predetermined value, transferring the historical key-value vector stored in the data block corresponding to the target request to the second memory, and clearing the data block corresponding to the target request; the second memory and the first memory correspond to different processors; allocating the first data block in the cleared third virtual group to the target request, and storing the target key-value vector in the first data block in the third virtual group.

6. The method according to claim 1, further comprising: The virtual group size, data block size, number of data blocks, and maximum number of requests processed are used as multiple hyperparameters corresponding to the model; Finding candidate parameter value combinations of the multiple hyperparameters; Using a test sample, a model evaluation is performed on the model set according to the candidate parameter value combination, and if the model evaluation result meets the performance requirements, the candidate parameter value combination is used as the target parameter value combination; according to the target virtual group size, target data block size, and target number of data blocks determined in the target parameter value combination, the first memory is divided into multiple virtual groups and multiple data blocks corresponding to the multiple virtual groups.

7. The method according to claim 6, wherein: The using the test sample to perform model evaluation on the model set according to the candidate parameter value combination includes: inputting the test sample into the model set according to the candidate parameter value combination to obtain a prediction result; and determining a model evaluation result based on difference information between the prediction result and the expected result.

8. The method according to claim 5, further comprising: Update the record information of the target request in the block table according to the storage information of the target key-value vector.

9. The method according to claim 5, further comprising: Calculating the target key-value vector corresponding to the text unit generated by the target request at the previous time step; Reading a history key-value vector corresponding to the target request from the first memory and / or the second memory; Concatenate the target key value vector and the historical key value vector to generate an input vector for the current time step; The model is used to generate a text unit corresponding to the current time step based on the input vector.

10. The method according to claim 9, wherein: The reading of the historical key-value vector corresponding to the target request from the first memory and / or the second memory includes: reading the historical key-value vector corresponding to the target request from the first memory and / or the second memory according to the record message of the target request in the block table.

11. The method according to claim 9, further comprising: Determining whether there is at least one text unit to be recalculated; If so, calculating at least one key-value vector corresponding to the at least one text unit; The step of concatenating the target key-value vector and the historical key-value vector to generate the input vector for the current time step includes: concatenating the target key-value vector, the at least one key-value vector, and the historical key-value vector to generate the input vector for the current time step.

12. A computing device comprising multiple memories and multiple processors; wherein: The multiple memories provide a first memory and a second memory; the first memory and the second memory are used to store key-value vectors; wherein the multiple memories store one or more computer program instructions for the first processor to call and execute, so as to implement the storage method according to any one of claims 1 to 11.

13. A computer-readable storage medium storing a computer program, wherein: The computer program is executed by a computer to implement the storage method according to any one of claims 1 to 11.

14. A computer program product storing a computer program, wherein: The computer program is executed by a computer to implement the storage method according to any one of claims 1 to 11.

Images