Large model intermediate data storage system based on GPU expansion

Abstract

The invention relates to the technical field of data storage, in particular to a large model intermediate data storage system based on GPU expansion, the system comprises a main GPU chip, an expansion video memory, a database, a processor and a memory in which a computer program is stored, and when the computer program is executed by the processor, the following steps are realized: when a large model executes a reasoning task, the expansion video memory is used for storing the large model intermediate data; according to the method, KV cache data corresponding to a target user is stored in a main video memory with relatively high performance so as to ensure relatively high calculation efficiency when the KV cache data is applied, and when the quantity of output sub-data generated for the target user reaches a preset value, the KV cache data corresponding to the target user and the output sub-data output finally are moved to an extended video memory to be stored, so that the calculation efficiency is improved. The target user is determined again, so that when the input data of each user is processed, the corresponding KV cache data are all in the main video memory, and the storage efficiency of the video memory is improved when multiple GPUs are interconnected.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, in particular to a large model intermediate data storage system based on GPU extension. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large models with a large number of model parameters have shown excellent performance. When performing inference tasks, large models can use KV cache to speed up inference. However, KV cache requires a large amount of storage space.

[0003] Currently, one method of the prior art is to store KV cache in CPU memory and transfer it to GPU in batches as needed through PCIE for calculation. However, due to the bandwidth and latency limitations of PCIE, it cannot effectively support the calculation supply requirements of GPU, that is, it affects the calculation efficiency of GPU. Another method of the prior art is to use multi-GPU interconnection to obtain larger video memory capacity. However, the increase in the number of multi-GPU chips greatly increases hardware costs, and the additional network hardware requirements caused by interconnection further increase the training and inference costs of large models. To overcome the above problems, an effective solution is to extend the video memory scheme, that is, to add an extension video memory to the main video memory, so that the KV cache data obtained by intermediate calculation can be stored in the main video memory and the extension video memory during the execution of the large model inference task. However, considering the use cost, the extension video memory scheme usually uses a small amount of storage with high efficiency and low capacity as the main video memory and a storage with low efficiency and high capacity as the extension video memory. However, due to the difference in storage efficiency of the video memory, the storage location of the intermediate data also affects the efficiency of the inference calculation. Therefore, how to optimize the storage of large model intermediate data to improve the storage efficiency of the video memory and thus improve the inference efficiency of the large model has become a problem to be solved. SUMMARY

[0004] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0005] A large model intermediate data storage system based on GPU extension, the system comprising: a main GPU chip, an extension video memory G2, a database, a processor, and a memory storing a computer program, wherein the main GPU chip corresponds to a main video memory G1, the database comprises weight data X of a large model, user identification sequence Y = [y1, y2, …, yN] and input data corresponding to each user, yn is the user identification of the nth user, n is an integer in the range of [1, N], N is the number of user identifications in Y, yn corresponds to zn, zn is the input data corresponding to the nth user, and the processor is connected with the main GPU chip, the extension video memory G2, the database and the memory. n , …, y N ] and input data corresponding to each user, y n is the user identification of the nth user, n is an integer in the range of [1, N], N is the number of user identifications in Y, y n corresponds to z n , zn For the input data of the nth user, the capacity D(G1) of the main memory G1 is greater than the amount D(X) of the weight data X stored in the main memory G1, when the computer program is executed by the processor, the following steps are implemented:

[0006] S101, at a preset time point t, determine the user corresponding to the user identification at the front of the user identification sequence Y as a target user.

[0007] S102, receive the input data of the target user, and calculate the KV cache data corresponding to the target user according to the weight data X and the input data of the target user, and store the KV cache data corresponding to the target user in the main memory G1.

[0008] S103, take a preset first identifier as reference data.

[0009] S104, initialize the output sub-data identifier i = 1.

[0010] S105, calculate the Query vector, Key vector and Value vector corresponding to the reference data according to the reference data and the weight data X.

[0011] S106, calculate the ith output sub-data O i .

[0012] S107, update the KV cache data corresponding to the target user according to the Key vector and the Value vector corresponding to the reference data.

[0013] S108, update the reference data with the output sub-data O i , and update i = i + 1.

[0014] S109, return to execute steps S105 to S108 until i = mar or the output sub-data O i is a preset second identifier, execute step S110, and mar is a preset output data amount threshold corresponding to the target user.

[0015] S110, take the output sub-data O i as historical data, take the KV cache data corresponding to the target user as historical cache data, and move the historical data and the historical cache data from the main memory G1 to the extended memory G2.

[0016] S111, delete the user identifier corresponding to the target user from the user identifier sequence Y, and update the user identifier sequence.

[0017] S112, update the preset time point t = t + T, return to execute steps S101 to S111 until the user identifier sequence Y is an empty sequence, and T is the preset interval duration.

[0018] Compared with the prior art, the present invention has significant advantages. Through the above technical solution, the large model intermediate data storage system based on GPU extension provided by the present invention achieves considerable technological progress and practicality, and has broad industrial application value. It has at least the following advantages:

[0019] This invention provides a large model intermediate data storage system based on GPU extension. The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes the weight data X of the large model and the user identifier sequence Y = [y1, y2, ..., y...]. n , ..., y N Given the input data for each user, when the computer program is executed by the processor, the following steps are implemented: At a preset time point t, the user corresponding to the first user identifier in the user identifier sequence Y is identified as the target user; the input data of the target user is received; based on the weight data X and the input data of the target user, the KV cache data corresponding to the target user is calculated; the KV cache data corresponding to the target user is stored in the main video memory G1; using a preset first identifier as reference data, the Query vector, Key vector, and Value vector corresponding to the reference data are calculated iteratively based on the reference data and the weight data X; based on the Query vector, Key vector, Value vector corresponding to the reference data and the KV cache data corresponding to the target user, the i-th output sub-data O is calculated. i Based on the key and value vectors corresponding to the reference data, update the KV cache data corresponding to the target user to output sub-data O. i The steps to update the reference data continue until i = mar or the sub-data O is output. i The second identifier is preset, and the output sub-data O will be used. i As historical data, the KV cache data corresponding to the target user is used as historical cache data. The historical data and historical cache data are moved from the main video memory G1 to the extended video memory G2. The user identifier corresponding to the target user is deleted from the user identifier sequence Y. The user identifier sequence is updated. The preset time point t = t + T is updated. The process is repeated until the user identifier sequence Y is an empty sequence.

[0020] It can be seen that when a large model performs an inference task, the key-value cache data corresponding to the target user is first stored in the high-performance main video memory to ensure high computational efficiency when applying the key-value cache data. When the number of output sub-data generated for the target user reaches a preset value, the key-value cache data corresponding to the target user and the final output sub-data are moved to the extended video memory for storage, and then the target user is re-determined. This ensures that when the input data of each user is processed, the corresponding key-value cache data is in the main video memory, thereby improving the storage efficiency of the video memory when multiple GPUs are interconnected. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps implemented when a computer program is executed by a processor in a GPU-based large model intermediate data storage system provided in an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] This invention provides a large model intermediate data storage system based on GPU extension. The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes the weight data X of the large model and the user identifier sequence Y = [y1, y2, ..., y...]. n , ..., y N ] and the input data corresponding to each user, y n Let be the user identifier of the nth user, where n is an integer in the range [1, N], and N is the number of user identifiers in Y. n Corresponding to z n , z n For the input data of the nth user, the capacity D(G1) of the main video memory G1 is greater than the amount D(X) of the weight data X. The weight data X is stored in the main video memory G1. See [link to relevant documentation]. Figure 1This is a flowchart illustrating the steps implemented when a computer program is executed by a processor in a large model intermediate data storage system based on GPU extension, as provided in an embodiment of the present invention. When the computer program is executed by the processor, the following steps are implemented:

[0025] S101, at a preset time point t, the user corresponding to the first user identifier in the user identifier sequence Y is identified as the target user;

[0026] S102, receive the input data of the target user, calculate the KV cache data corresponding to the target user according to the weight data X and the input data of the target user, and store the KV cache data corresponding to the target user in the main video memory G1;

[0027] S103, using a preset first identifier as reference data;

[0028] S104, Initialize the output sub-data identifier i = 1;

[0029] S105, Based on the reference data and weight data X, calculate the Query vector, Key vector and Value vector corresponding to the reference data;

[0030] S106, calculate the i-th output sub-data O based on the Query vector, Key vector, Value vector corresponding to the reference data and the KV cache data corresponding to the target user. i ;

[0031] S107, Update the KV cache data corresponding to the target user based on the Key vector and Value vector corresponding to the reference data;

[0032] S108, output sub-data O i Update the reference data, and update i = i + 1;

[0033] S109, return to steps S105 to S108 until i = mar or output sub-data O. i For the preset second identifier, execute step S110, where mar is the preset output data volume threshold corresponding to the target user;

[0034] S110 will output sub-data O i As historical data, the KV cache data corresponding to the target user is used as historical cache data. The historical data and historical cache data are moved from the main video memory G1 to the extended video memory G2.

[0035] S111, delete the user identifier corresponding to the target user from the user identifier sequence Y, and update the user identifier sequence;

[0036] S112, update the preset time point t=t+T, return to execute steps S101 to S111 until the user identification sequence Y is an empty sequence, and T is a preset interval length.

[0037] The GPU extension scheme can be implemented using a main graphics memory with high storage performance and an extension graphics memory with low storage performance. The main graphics memory can use HBM storage particles, and the extension graphics memory can use storage particles with lower cost or lower bandwidth than HBM storage particles.

[0038] The capacity D(G1) of the main graphics memory G1 is greater than the amount D(X) of the weight data X as a prior condition, that is, the main graphics memory G1 can store data used in the inference process of the large model in addition to the weight data of the large model. The data used in the inference process of the large model can include user input data, intermediate calculation result data, KV cache data, etc. The embodiment mainly performs storage optimization for KV cache data.

[0039] The user identification sequence Y can be used to represent the order of processing user input data. After the database receives the input data of a new user, the user identification sequence Y can be updated, that is, the user identification of the new user is set to y N+1 , which is inserted after the last position of the user identification sequence Y, and N=N+1. It should be noted that the embodiment can be regarded as only optimizing storage for single-thread. In actual use, the inference task of the large model is usually executed in multi-thread, and each thread can apply the embodiment for storage optimization.

[0040] The preset time point t can be the time point at which the inference task is started. At this time, since the user identification sequence Y can be used to represent the order of processing user input data, the user corresponding to the earliest user identification in the user identification sequence Y is determined as the target user, that is, the user to execute the inference task.

[0041] In the inference process of the large model, the KV cache data corresponding to the target user is stored in the main graphics memory G1, so that the access of the KV cache data is realized with high storage efficiency.

[0042] The first identifier can be a starting symbol of output data generation, for example, represented by “[s]”. The calculation of the Query vector, the Key vector, and the Value vector uses the conventional calculation process of the attention mechanism, which will not be described here.

[0043] In the large model inference process, tokens are generated one by one, that is, output sub-data is output, and each time the generated output sub-data is used as the input for the next generation. In addition, the Key vector and the Value vector calculated each time are updated to the KV cache data by splicing, so as to be directly used in subsequent calculations.

[0044] The second identifier can be a terminator of output data generation, for example, can be represented by "[e]". When i = mar, it means that the target user has generated a preset number of tokens. When the output sub-data O i The second identifier is a preset second identifier, which indicates that the inference task for the target user has been completed. When the above two cases occur, the inference task of the target user is suspended, the last generated output sub-data of the target user is used as historical data, and the KV cache data corresponding to the target user is used as historical cache data. The main memory is transferred to the extended memory for storage. In this embodiment, a preset output data quantity threshold is set for the single inference of the target user, so that the single inference of the target user does not have to complete the inference task, but allows to generate part of the tokens, so as to meet the interaction habit while providing inference tasks for more users. For example, 20 tokens of output data need to be provided for user A, and the generation efficiency of tokens is 20 tokens / s. At this time, 20 tokens of output data does not need to be directly provided for user A after 1 second, but 10 tokens of output data can be generated for user A after 1 second, and 10 tokens of output data can be provided for another user B, so as to cooperate with the output speed of the interaction display interface, for example, the output speed can be 1 second 10 tokens to display the inference result. Therefore, 20 tokens are generated directly in 1 second, and 20 tokens are generated in two seconds. The reading experience of the user is not affected, but the latter can provide inference services for two users at the same time in the first second, which is equivalent to increasing the concurrency of the inference task.

[0045] After the historical cache data and the historical data of the target user are transferred to the extended memory, the user identifier corresponding to the target user is deleted from the user identifier sequence Y, the user identifier sequence is updated, the preset time point t = t + T is updated, and steps S101 to S111 are returned to be executed. That is, the target user is determined again, and the inference process is executed until the user identifier sequence Y is empty, that is, the inference tasks of all users are completed. In this embodiment, the preset interval T can be set to 0.5 seconds. The interval T can be determined by the average execution time of a single user's single inference service.

[0046] In a specific embodiment, the historical data and the historical cache data are moved from the main memory G1 to the extended memory G2, including:

[0047] Move the historical data and the historical cache data from the main graphics memory G1 to the extended graphics memory G2, and take the time point when the moving is completed as an initial time point;

[0048] Add the initial time point and the preset recall duration corresponding to the target user to determine a sum result as a recall time point corresponding to the target user, and the recall time point is used to update the user identifier corresponding to the target user to the user identifier sequence Y.

[0049] The recall duration can refer to an interval duration for providing reasoning service for the target user again. The time point when the intermediate data is moved is taken as a time point when the current reasoning service ends, that is, the initial time point. After the initial time point and the preset recall duration corresponding to the target user are added, a time point for starting the next reasoning service is obtained, that is, the recall time point.

[0050] In a specific implementation, the step S112 includes:

[0051] In the step S201, the preset time point t is updated to t+T, and a target time period [t, t+T] is determined.

[0052] In the step S202, the users with the recall time points in the target time period are assigned with recall identifiers, and the user identifiers corresponding to the users with the recall identifiers are sequentially inserted into the first position of the user identifier sequence Y in the order from the rear to the front of the recall time points, and the user identifier sequence Y is updated.

[0053] In the step S203, the steps S101 to S111 are returned to be executed until the user identifier sequence Y is an empty sequence.

[0054] The preset time point is initially assumed to be 0, and the updated preset time point is T, and the target time period is [T, 2T].

[0055] If the recall time point is in the target time period, it is indicated that the reasoning service needs to be provided for the users corresponding to the recall time points in the target time period. The users with the recall time points in the target time period are assigned with the recall identifiers, and the recall identifier can be represented by 1. Since there can be multiple users with the recall time points in the target time period, the user identifiers corresponding to the users with the recall identifiers are sequentially inserted into the first position of the user identifier sequence Y in the order from the rear to the front of the recall time points, and the user identifier sequence Y is updated. At this time, the user identifier corresponding to the user in the front of the recall time point is in the first position of the user identifier sequence Y, thereby ensuring the use experience of the user on the reasoning service.

[0056] In a specific implementation, the step S202 includes:

[0057] In the step S301, the users with the recall time points in the target time period are assigned with the recall identifiers.

[0058] In step S302, if the history data of any user with the recall mark in the extended memory G2 is the second identifier and the user has not received the updated input data before the preset time point, the recall mark of the user is changed to a retention mark, and after the user with the retention mark updates the input data, the corresponding retention mark of the user is changed to a recall mark.

[0059] In step S303, according to the order from the back to the front of the recall time points, the user identifier corresponding to each user with the recall mark is sequentially inserted before the first position of the user identifier sequence Y, and the user identifier sequence Y is updated.

[0060] When the history data of the user with the recall mark in the extended memory G2 is the second identifier and the user has not received the updated input data before the preset time point, it indicates that the user may suspend or terminate the use of the reasoning service at this time, and thus it is not necessary to provide the reasoning service for the user in the target time period, and the recall mark of the user is changed to a retention mark. When the user with the retention mark updates the input data, the corresponding retention mark of the user is changed to a recall mark. It should be noted that the recall time point of the user can not be changed. After the user updates the input data, the user identifier sequence is updated according to the recall time point, and the user identifier sequence is as far forward as possible, so that the reasoning service can be provided for the user.

[0061] In a specific embodiment, after step S101, when the computer program is executed by the processor, the following steps are also implemented:

[0062] In step S401, if the target user does not have the recall mark, step S102 is continued to be executed.

[0063] In step S402, if the target user has the recall mark, and the history data stored in the extended memory G2 of the target user is not the second identifier, the history data is taken as the reference data, and the history cache data stored in the extended memory G2 of the target user is taken as the KV cache data corresponding to the target user, and step S104 is executed.

[0064] If the target user does not have the recall mark, it indicates that the target user does not store the history cache data and the history data in the extended memory, and the input data of the target user is taken as new input data for processing.

[0065] If the target user has the recall identifier, and the historical data stored by the target user in the extended memory G2 is not the second identifier, it indicates that the current inference service of the target user is in an unfinished state, and the inference service needs to be continued for the target user. At this time, the historical data is used as the reference data, and the historical cache data stored by the target user in the extended memory G2 is used as the KV cache data corresponding to the target user, so as to continue to provide the unfinished inference service for the target user.

[0066] In a specific embodiment, after step S101, when the computer program is executed by the processor, the following steps are also implemented:

[0067] In step S403, if the target user has the recall identifier, and the historical data stored by the target user in the extended memory G2 is the second identifier, and the target user updates the input data before the preset time point, the input data of the target user is received, the KV cache data corresponding to the target user is calculated according to the weight data X and the input data of the target user, the historical cache data stored by the target user in the extended memory G2 is used to update the KV cache data corresponding to the target user, the KV cache data corresponding to the target user is stored in the main memory G1, and step S103 is executed.

[0068] If the target user has the recall identifier, and the historical data stored by the target user in the extended memory G2 is the second identifier, and the target user updates the input data before the preset time point, it indicates that the inference service of the target user has been completed, but there is a new inference task. At this time, in order to enable the large model to utilize the context information of the target user, after calculating the new KV cache data according to the new input data, the historical cache data is spliced with the newly calculated KV cache data to update the KV cache data.

[0069] In a specific embodiment, when the computer program is executed by the processor, the following steps are also implemented:

[0070] The user priority corresponding to the target user is obtained.

[0071] The user priority corresponding to the target user is mapped to the preset output data quantity threshold corresponding to the target user using the mapping relationship between the user priority and the output data quantity threshold.

[0072] The user priority can be determined according to actual conditions, for example, by user payment conditions, user use frequency conditions, etc., so that customers with higher priority can obtain better inference service experience.

[0073] In an embodiment, the preset recall duration can also be adjusted according to the user priority.

[0074] In a specific embodiment, mar is set to 10.

[0075] In general, the output data volume thresholds of all users are the same by default, and in this embodiment, mar is set to 10. The implementer can adjust the value of mar according to the actual situation.

[0076] In this embodiment, when the large model performs the inference task, the KV cache data corresponding to the target user is first stored in the main memory with high performance, so as to ensure that the calculation efficiency when the KV cache data is applied is high. When the number of output sub-data generated for the target user reaches the preset value, the KV cache data corresponding to the target user and the last output output sub-data are moved to the extended memory storage, and the target user is re-determined, so that the input data of each user is processed, and the KV cache data corresponding thereto is in the main memory, thereby improving the storage efficiency of the memory when the multiple GPUs are interconnected.

[0077] Although some specific embodiments of the present application have been described in detail through examples, those skilled in the art should understand that the above examples are only for illustration, not for limiting the scope of the present application. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the present application. The scope of the present application is defined by the appended claims.

Claims

1. A large model intermediate data storage system based on GPU extension, characterized in that, The system includes: a main GPU chip, extended video memory G2, a database, a processor, and a memory storing computer programs. The main GPU chip corresponds to the main video memory G1. The database includes weight data X of a large model and a user identifier sequence Y = [y1, y2, ..., y...]. n , ..., y N ] and the input data corresponding to each user, y n Let be the user identifier of the nth user, where n is an integer in the range [1, N], and N is the number of user identifiers in Y. n Corresponding to z n , z n For the input data of the nth user, the capacity D(G1) of the main video memory G1 is greater than the amount D(X) of the weight data X, and the weight data X is stored in the main video memory G1. When the computer program is executed by the processor, the following steps are implemented: S101, at a preset time point t, the user corresponding to the first user identifier in the user identifier sequence Y is determined as the target user; S102, receive the input data of the target user, calculate the KV cache data corresponding to the target user according to the weight data X and the input data of the target user, and store the KV cache data corresponding to the target user in the main video memory G1; S103, using a preset first identifier as reference data; S104, Initialize the output sub-data identifier i = 1; S105, Calculate the Query vector, Key vector and Value vector corresponding to the reference data based on the reference data and the weight data X; S106, calculate the i-th output sub-data O based on the Query vector, Key vector, Value vector corresponding to the reference data and the KV cache data corresponding to the target user. i ; S107, Update the KV cache data corresponding to the target user according to the Key vector and Value vector corresponding to the reference data; S108, output sub-data O i Update the reference data, and update i = i + 1; S109, return to steps S105 to S108 until i = mar or the output sub-data O. i If the second identifier is preset, execute step S110, where mar is the preset output data volume threshold corresponding to the target user; S110 will output sub-data O i As historical data, the KV cache data corresponding to the target user is used as historical cache data. The historical data and the historical cache data are moved from the main video memory G1 to the extended video memory G2. S111, delete the user identifier corresponding to the target user from the user identifier sequence Y, and update the user identifier sequence; S112, update the preset time point t = t + T, return to execute steps S101 to S111 until the user identifier sequence Y is an empty sequence, and T is the preset interval duration.

2. The large model intermediate data storage system based on GPU extension according to claim 1, characterized in that, Moving the historical data and the historical cache data from the main video memory G1 to the extended video memory G2 includes: The historical data and the historical cache data are moved from the main video memory G1 to the extended video memory G2, and the time point when the move is completed is used as the initial time point. The initial time point and the preset recall duration corresponding to the target user are added together to determine the recall time point corresponding to the target user. The recall time point is used to update the user identifier of the corresponding target user into the user identifier sequence Y.

3. The large model intermediate data storage system based on GPU extension according to claim 2, characterized in that, Step S112 includes: Step S201: Update the preset time point t = t + T, and determine the target time period [t, t + T]. Step S202: Assign recall identifiers to users whose recall time points fall within the target time period. In order from the last to the first of the recall time points, insert the user identifiers corresponding to each user with a recall identifier into the first position of the user identifier sequence Y, and update the user identifier sequence Y. Step S203: Return to steps S101 to S111 until the user identifier sequence Y is an empty sequence.

4. The large model intermediate data storage system based on GPU extension according to claim 3, characterized in that, Step S202 includes: Step S301: Users whose recall time falls within the target time period are assigned a recall identifier; Step S302: If any user with a recall identifier has a second identifier in the historical data of the extended video memory G2 and has not received updated input data from the user before the preset time point, the recall identifier of the user is changed to a retention identifier. After the user with a retention identifier updates the input data, the corresponding retention identifier is changed to a recall identifier. Step S303: According to the recall time point from back to front, insert the user identifiers corresponding to each user with the recall identifier into the first position of the user identifier sequence Y in sequence, and update the user identifier sequence Y.

5. The large model intermediate data storage system based on GPU extension according to claim 4, characterized in that, After step S101, when the computer program is executed by the processor, the following steps are also performed: Step S401: If the target user does not have a recall identifier, then proceed to step S102. Step S402: If the target user has a recall identifier and the historical data stored by the target user in the extended video memory G2 is not a second identifier, then the historical data is used as reference data, and the historical cache data stored by the target user in the extended video memory G2 is used as the KV cache data corresponding to the target user, and the process jumps to step S104.

6. The GPU-extended large model intermediate data storage system according to claim 5, characterized in that, After step S101, when the computer program is executed by the processor, the following steps are also performed: Step S403: If the target user has a recall identifier, and the historical data stored by the target user in the extended video memory G2 is a second identifier, and the target user updated the input data before the preset time point, then the input data of the target user is received, the KV cache data corresponding to the target user is calculated according to the weight data X and the input data of the target user, the KV cache data corresponding to the target user is updated using the historical cache data stored by the target user in the extended video memory G2, the KV cache data corresponding to the target user is stored in the main video memory G1, and the process jumps to step S103.

7. The large model intermediate data storage system based on GPU extension according to claim 1, characterized in that, When the computer program is executed by the processor, the following steps are also performed: Obtain the user priority corresponding to the target user; Using the mapping relationship between user priority and output data volume threshold, the user priority corresponding to the target user is mapped to the preset output data volume threshold corresponding to the target user.

8. The large model intermediate data storage system based on GPU extension according to claim 1, characterized in that, Mar is set to 10.

Images