Data storage system, method and device, electronic equipment and storage medium

Abstract

The invention discloses a data storage system, method and device, electronic equipment and a storage medium, the data storage system comprises a global scheduling module, at least one reasoning module and a storage module, and the global scheduling module is used for scheduling the at least one reasoning module when it is determined that cache data associated with a received reasoning request exists in the storage module. Determining a target reasoning module from the at least one reasoning module based on the load information of the at least one reasoning module and the acquired transmission cost of the cache data to the at least one reasoning module; the at least one reasoning module is used for loading cache data corresponding to the reasoning request from the storage module based on the reasoning request, and performing model reasoning based on the cache data; the storage module is used for storing the target cache data sent by the reasoning module and managing the cache data and the target cache data; by optimizing the resource management and scheduling mechanism of the cache data, the overall utilization rate of the cache resources and the reasoning service performance are improved, and the concurrent bearing capacity and the response efficiency of the reasoning service are enhanced.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to a data storage system, method and apparatus, electronic device and storage medium. Background Technology

[0002] With the development of artificial intelligence technology, key-value caching storage technology plays a crucial role in fields such as large language model inference services, distributed intelligent computing power scheduling, and intelligent agent infrastructure construction. Key-value caching storage technology is widely used in the performance optimization and cost control of various large language model inference services.

[0003] In related technologies, key-value caches are constructed by using key-value matrices in the cache attention mechanism to avoid repeated calculations of historical tokens, thereby achieving efficient autoregressive inference for large language models. However, traditional key-value caching schemes suffer from low overall resource utilization due to imperfect cache resource management and scheduling mechanisms, resulting in limited inference service performance. Summary of the Invention

[0004] Based on this, this disclosure provides a data storage system, method, apparatus, electronic device, and storage medium to solve the problems in related technologies. By optimizing the resource management and scheduling mechanism of cached data, it improves the overall utilization rate of cached resources and the performance of inference services, and enhances the concurrent carrying capacity and response efficiency of inference services.

[0005] According to a first aspect of this disclosure, a data storage system is provided, the system comprising: a global scheduling module, at least one inference module, and a storage module, wherein... The global scheduling module is used to determine a target inference module from the at least one inference module based on the load information of the at least one inference module and the transmission cost of obtaining the cached data to the at least one inference module when it is determined that there is cached data associated with the received inference request in the storage module, and to schedule the inference request to the target inference module. The at least one inference module is configured to respond to receiving the inference request sent by the global scheduling module, load the cached data corresponding to the inference request from the storage module based on the inference request, perform model inference based on the cached data, and send the target cached data generated during the model inference process to the storage module; The storage module is used to store the target cache data sent by the inference module, and manage the cache data and the target cache data according to a predefined structure, so as to provide the cache data access service to the at least one inference module.

[0006] In some embodiments of this disclosure, the global scheduling module is further configured to: In response to receiving the inference request, the system queries the metadata table to determine whether the cached data associated with the inference request exists in the storage module. If it is determined that cached data associated with the inference request exists in the storage module, the load information of the at least one inference module is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; The transmission cost of transferring the cached data from the storage module to the at least one inference module is compared with the computation cost of the at least one inference module to obtain the comparison result; Based on the load information and the comparison results, the target inference module is determined from the at least one inference module.

[0007] In some embodiments of this disclosure, the global scheduling module is further configured to: If the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after the current inference request is processed, new inference requests are rejected and the module enters a sleep mode.

[0008] In some embodiments of this disclosure, the global scheduling module is further configured to: If it is determined that the target inference module has failed while processing the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost. The cached data generated in the faulty inference module that has failed, as well as the target cached data corresponding to the inference request, are migrated to the new inference module through a preloading mechanism. The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

[0009] In some embodiments of this disclosure, the inference module is further configured to: During the sequence filling stage of model inference, the cache data generated by each layer of the model is transferred to the host memory of the storage module; During the decoding and generation phase of model inference, at least one cached data required for the current computation is saved to the processor's video memory, and the cached data not yet used for computation is saved in the host memory; wherein, the inference module is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

[0010] In some embodiments of this disclosure, the global scheduling module is further configured to: The at least one inference module is divided into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

[0011] In some embodiments of this disclosure, the global scheduling module is further configured to: In the sequence filling stage of processing the long sequence inference request, all inference modules in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result; In the decoding and generation stage of processing the initial processing results, the number of inference modules is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0012] In some embodiments of this disclosure, the global scheduling module is further configured to: If it is determined that the parallelism of the parallel processing group has decreased, the cached data corresponding to the inference module in the parallel processing group is saved; wherein, the parallelism is the number of inference modules; If the parallelism of the parallel processing group is increased, the cached data stored in the inference module is sent to the new inference module for reuse.

[0013] According to a second aspect of this disclosure, a data storage method is provided, comprising: In response to receiving an inference request, determine whether there is cached data in the storage module associated with the inference request based on the inference request; If it is determined that there is cached data associated with the inference request, obtain the load information of at least one inference module and the transmission cost of the cached data to the at least one inference module; Based on the load information and the transmission cost, a target inference module is determined from the at least one inference module; The inference request is scheduled to the target inference module.

[0014] In some embodiments of this disclosure, determining the target inference module from the at least one inference module based on the load information and the transmission cost includes: In response to receiving the inference request, if it is determined in the metadata table that there is cached data associated with the inference request in the storage module, the load information of the at least one inference module is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; The transmission cost of transferring the cached data from the storage module to the at least one inference module is compared with the computation cost of the at least one inference module to obtain the comparison result; Based on the load information and the comparison results, the target inference module is determined from the at least one inference module.

[0015] In some embodiments of this disclosure, after scheduling the inference request to the target inference module, the method further includes: If the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after the current inference request is processed, new inference requests are rejected and the module enters a sleep mode.

[0016] In some embodiments of this disclosure, after scheduling the inference request to the target inference module, the method further includes: If it is determined that the target inference module has failed while processing the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost. The cached data generated in the faulty inference module that has failed, as well as the target cached data corresponding to the inference request, are migrated to the new inference module through a preloading mechanism. The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

[0017] In some embodiments of this disclosure, scheduling the inference request to the target inference module includes: The at least one inference module is divided into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

[0018] In some embodiments of this disclosure, the step of dynamically adjusting the number of inference modules being processed in the current round of the parallel processing group based on computing power resources and memory usage requirements during the long sequence inference request processing includes: In the sequence filling stage of processing the long sequence inference request, all inference modules in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result; In the decoding and generation stage of processing the initial processing results, the number of inference modules is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0019] In some embodiments of this disclosure, after dividing the at least one inference module into a parallel processing group, the method further includes: If it is determined that the parallelism of the parallel processing group has decreased, the cached data corresponding to the inference module in the parallel processing group is saved; wherein, the parallelism is the number of inference modules; If the parallelism of the parallel processing group is increased, the cached data stored in the inference module is sent to the new inference module for reuse.

[0020] According to a third aspect of this disclosure, a data storage method is provided, comprising: In response to receiving an inference request, the corresponding cached data is loaded from the storage module based on the inference request; Model inference is performed based on the cached data; The target cache data generated during the model inference process is sent to the storage module so that the storage module can store it and reuse the inference request.

[0021] In some embodiments of this disclosure, the model inference based on the cached data includes: During the sequence filling stage of model inference, the cache data generated by each layer of the model is transferred to the host memory of the storage module; During the decoding and generation phase of model inference, at least one cached data required for the current computation is saved to the processor's video memory, while uncomputed cached data is saved in the host memory. The inference module is configured to overlap the computation operations of each layer with the communication operations that load the cached data required for the next layer from the host memory in time.

[0022] According to a fourth aspect of this disclosure, a data storage device is provided, comprising: The first determining unit is configured to, in response to receiving an inference request, determine whether there is cached data associated with the inference request in the storage module based on the inference request; The acquisition unit is configured to, upon determining the existence of cached data associated with the inference request, acquire load information of at least one inference module and acquire the transmission cost of the cached data to the at least one inference module; The second determining unit is used to determine a target inference module from the at least one inference module based on the load information and the transmission cost; The scheduling unit is used to schedule the inference request to the target inference module.

[0023] In some embodiments of this disclosure, the second determining unit includes: The acquisition module is configured to, in response to receiving the inference request, and if it is determined from the metadata table that the cached data associated with the inference request exists in the storage module, acquire the load information of the at least one inference module; wherein the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; A comparison module is used to compare the transmission cost of the cached data from the storage module to the at least one inference module with the computation cost of the at least one inference module, and obtain a comparison result; A determination module is used to determine the target inference module from the at least one inference module based on the load information and the comparison results.

[0024] In some embodiments of this disclosure, the apparatus further includes: The switching unit is configured to, after the scheduling unit schedules the inference request to the target inference module, and if it is determined that the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after processing the currently processed inference request, refuse to receive new inference requests and enter a sleep mode.

[0025] In some embodiments of this disclosure, the apparatus further includes: A creation unit is configured to, after the scheduling unit schedules the inference request to the target inference module, and if it is determined that the target inference module has failed in processing the inference request, create a new inference module based on the load information and the transmission cost through an elastic scaling mechanism. The migration unit is used to migrate the cached data generated in the faulty inference module that has failed and the target cached data corresponding to the inference request to the new inference module through a preloading mechanism. The first sending unit is used to resend the inference request and a portion of the text results generated by the fault inference module to the new inference module to continue completing the inference request.

[0026] In some embodiments of this disclosure, the scheduling unit includes: A partitioning module is used to divide the at least one inference module into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; The adjustment module is used to dynamically adjust the number of inference modules being processed in the current round of the parallel processing group according to the computing power resources and memory usage requirements during the long sequence inference request processing.

[0027] In some embodiments of this disclosure, the adjustment module includes: The processing submodule is used to process the long sequence inference request in parallel using all the inference modules in the parallel processing group during the sequence filling stage of processing the long sequence inference request, and to obtain the initial processing result. A submodule is added to dynamically increase the number of inference modules during the decoding and generation stage of processing the primary processing results, based on the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0028] Some embodiments of this disclosure include: A first storage module is configured to, after the partitioning module has divided the at least one inference module into a parallel processing group, and upon determining that the parallelism of the parallel processing group has decreased, store the cached data corresponding to the inference module in the parallel processing group; wherein the parallelism is the number of inference modules; The sending module is used to send the cached data stored in the inference module to a new inference module for reuse when it is determined that the parallelism of the parallel processing group has increased.

[0029] According to a fifth aspect of this disclosure, a data storage device is provided, comprising: A loading unit is used to load corresponding cached data from the storage module based on the received inference request. The inference unit is used to perform model inference based on the cached data; The second sending unit is used to send the target cache data generated during the model inference process to the storage module so that the storage module can store it and reuse the inference request.

[0030] In some embodiments of this disclosure, the inference unit includes: The transmission module is used to transfer the cached data generated by each layer of the model to the host memory of the storage module during the sequence filling stage of model inference. The second storage module is used to save at least one cached data currently being computed to the processor's video memory and save uncomputed cached data to the host memory during the decoding and generation phase of model inference; wherein the inference module is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

[0031] According to a sixth aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the second or third aspect embodiments described above.

[0032] According to a seventh aspect embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the methods described in the second or third aspect embodiments described above.

[0033] According to an eighth aspect embodiment of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in the foregoing second or third aspect embodiments.

[0034] In summary, the data storage system, method, apparatus, electronic device, and storage medium provided in this disclosure include: a global scheduling module, at least one inference module, and a storage module. The global scheduling module, upon determining that cached data associated with a received inference request exists in the storage module, determines a target inference module from at least one inference module based on the load information of the at least one inference module and the transmission cost of obtaining cached data to the at least one inference module, and schedules the inference request to the target inference module. At least one inference module is used to load the cached data corresponding to the inference request from the storage module based on the inference request, perform model inference based on the cached data, and send the target cached data generated during the model inference process to the storage module. The storage module stores the target cached data sent by the inference module and manages the cached data and target cached data according to a predefined structure. By optimizing the resource management and scheduling mechanism of cached data, the overall utilization rate of cached resources and the performance of inference services are improved, enhancing the concurrent carrying capacity and response efficiency of inference services.

[0035] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0036] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is a schematic diagram of the structure of a data storage system provided in an embodiment of the present disclosure; Figure 2 This is an architecture diagram of a data storage system provided in an embodiment of the present disclosure; Figure 3 This is a schematic diagram illustrating the interaction between a cached data structure and an inference module provided in an embodiment of this disclosure; Figure 4 This is a schematic diagram of a cached data transmission parameter provided in an embodiment of this disclosure; Figure 5 A logical diagram of a cross-inference module provided in an embodiment of this disclosure; Figure 6 This is a schematic flowchart illustrating a data storage method provided in an embodiment of the present disclosure. Figure 7 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 8 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 9 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 10 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 11 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 12 This is a schematic flowchart illustrating a data storage method provided in an embodiment of the present disclosure. Figure 13 This is a schematic flowchart illustrating another data storage method provided in an embodiment of the present disclosure; Figure 14 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present disclosure; Figure 15 A schematic diagram of another data storage device provided in an embodiment of this disclosure; Figure 16 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present disclosure; Figure 17 A schematic diagram of another data storage device provided in an embodiment of this disclosure; Figure 18 A schematic block diagram of an example electronic device provided for embodiments of this disclosure. Detailed Implementation

[0037] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0038] The data storage system, method, apparatus, electronic device, and storage medium of embodiments of the present disclosure are described below with reference to the accompanying drawings.

[0039] Figure 1 This is a schematic diagram of the structure of a data storage system provided in an embodiment of the present disclosure.

[0040] like Figure 1 As shown, the system includes: A global scheduling module 11, at least one inference module 12, and a storage module 13, wherein, The global scheduling module 11 is used to determine a target inference module from the at least one inference module 12 based on the load information of the at least one inference module 12 and the transmission cost of obtaining the cached data to the at least one inference module 12 when it is determined that there is cached data associated with the received inference request in the storage module 13, and to schedule the inference request to the target inference module.

[0041] In some embodiments, an inference request refers to a task request initiated by a user that requires a pre-trained language model to execute, including but not limited to requests for tasks such as text generation, question-and-answer interaction, and text summarization; cached data refers to the matrix data (i.e., KVCache data) corresponding to key vectors and value vectors generated by the pre-trained language model during inference, which serves to avoid repeated calculations of historical tokens. A token is the smallest unit of text processed by the large language model, including Chinese characters, English words, punctuation marks, etc.; after receiving an inference request, the global scheduling module 11 confirms whether there is associated cached data in the storage module 13 through a preset matching method. The matching method includes, but is not limited to, searching based on the text features and task identifiers of the inference request. Text features can be selected from keywords, prompt word fragments, etc., in the inference request; the load information of at least one inference module 12 is data reflecting the current processing capacity of the module, including but not limited to the number of inference requests being processed, processor utilization, and cache resource usage ratio. The load information is obtained through real-time communication between the global scheduling module 11 and each inference module 12. The transmission cost of acquiring cached data to the inference module 12 refers to the relevant quantitative indicators consumed in transmitting cached data from the storage module 13 to the inference module 12, including but not limited to transmission time and communication link occupancy. The transmission time can be calculated from the size of the cached data and the transmission link bandwidth, and the communication link occupancy can be determined by monitoring the data traffic during transmission. When determining the target inference module, the global scheduling module 11 adopts a comprehensive evaluation method, screening based on load information and transmission cost. For example, it first screens inference modules whose load information is within a preset reasonable range, and then selects the module with the lowest transmission cost from this range as the target inference module; or it uses preset weights to quantify the load information and transmission cost, and selects the inference module with the best comprehensive result as the target inference module. After determining the target inference module, the global scheduling module 11 sends the inference request and the associated cached data identifier to the target inference module to complete the request scheduling. It should be noted that the above examples are only illustrative and do not limit the specific content.

[0042] The above structure allows for the allocation of inference tasks based on the actual load of the inference module and the efficiency of cache transmission, avoiding excessive load on a single inference module and reducing resource consumption during cache data transmission, thus supporting the orderly processing of inference requests.

[0043] The at least one inference module 12 is configured to respond to receiving the inference request sent by the global scheduling module 11, load the cached data corresponding to the inference request from the storage module 13 based on the inference request, perform model inference based on the cached data, and send the target cached data generated during the model inference process to the storage module 13.

[0044] In some embodiments, the inference module 12 receives an inference request and a cached data identifier sent by the global scheduling module 11 through a preset communication interface. The communication interface can be implemented via, but is not limited to, an API interface or an internal data transmission channel. The inference module 12 sends a data loading request to the storage module 13 based on the cached data identifier. The request includes the cached data identifier and its own module identifier. The storage module 13 locates the corresponding cached data based on the inference request and transmits it to the inference module 12. The data transmission method includes, but is not limited to, data transmission and network transmission. After the cached data is loaded, the inference module 12 verifies the data integrity. The verification method includes, but is not limited to, checksum verification and data length comparison. The inference module 12 is internally deployed... There is a pre-set large language model. The model inference process refers to using the key vector and value vector matrix corresponding to historical tokens in the loaded cache data, combined with the input information of the current inference step, to complete the calculation through the attention mechanism, generate the output result of the current step, and gradually form a complete inference result. The target cache data is the KVCache data added during the inference process, that is, the new key vector and value vector matrix generated by the current inference step. After generating the target cache data, the inference module 12 adds an association identifier to it. The association identifier includes, but is not limited to, the inference request identifier, the generation time information, the token sequence identifier, etc., and then sends the target cache data to the storage module 13 through the data transmission channel.

[0045] The above structure allows for the direct reuse of existing cached data in the storage module, avoiding repeated calculations of historical tokens and shortening the time consumed in the inference process. At the same time, newly added target cached data can be stored back to the storage module, providing reusable cached resources for other related inference requests and realizing the cyclical accumulation of cached data.

[0046] The storage module 13 is used to store the target cache data sent by the inference module 12, and manage the cache data and the target cache data according to a predefined structure, so as to provide the cache data access service to the at least one inference module 12.

[0047] In some embodiments, the storage medium of the storage module 13 includes, but is not limited to, GPU memory, CPU memory, local storage devices, and network storage devices. After receiving the target cache data sent by the inference module 12, the storage module 13 stores the data to the corresponding medium according to preset storage rules. The storage rules are determined based on the data access requirements. For example, frequently accessed cache data is stored in GPU memory or CPU memory with faster access speed, while frequently accessed cache data is stored in local storage devices or network storage devices. The predefined structure is a classification structure set up to achieve orderly management of cache data, including but not limited to structures divided by model identifier, data type, token sequence range, etc. For example, cache data corresponding to different preset language models can be distinguished by model identifier, key vector data and value vector data can be distinguished by data type, and different... The storage module 13 manages the cached data corresponding to the text fragments. This includes data index creation, data storage maintenance, and data lookup and location. Data index creation is based on the association identifiers of the cached data to build index information. Data storage maintenance refers to monitoring the status of the stored cached data to ensure the integrity of the data storage. Data lookup and location refers to quickly locating the cached data required by the inference module 12 based on the index information. The storage module 13 provides services to at least one inference module 12 through a preset access interface. The cached data loading request sent by the inference module 12 is received by the storage module 13 through the access interface. After the storage module 13 finds the corresponding cached data based on the association identifiers in the request, it transmits the data to the inference module 12 through this interface. The target cached data sent by the inference module 12 is also received by the storage module 13 through this interface and stored, thus achieving a closed loop for cached data access.

[0048] The above structure enables the orderly storage and efficient management of cached data, ensuring that the inference module 12 can quickly obtain the required cached data. At the same time, by adapting to multiple types of storage media, it takes into account both the speed and capacity requirements of cached data storage, improves the utilization efficiency of storage resources, and provides data support for the stable operation of the system as a whole.

[0049] In summary, the data storage system provided in this disclosure includes: a global scheduling module, at least one inference module, and a storage module. The global scheduling module is used to determine a target inference module from at least one inference module based on the load information of at least one inference module and the transmission cost of obtaining cached data to at least one inference module, when it is determined that cached data associated with the received inference request exists in the storage module. At least one inference module is used to load cached data corresponding to the inference request from the storage module based on the inference request and perform model inference based on the cached data. The storage module is used to store the target cached data sent by the inference module and manage the cached data and the target cached data. By optimizing the resource management and scheduling mechanism of cached data, the overall utilization rate of cached resources and the performance of inference services are improved, and the concurrent carrying capacity and response efficiency of inference services are enhanced.

[0050] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: In response to receiving the inference request, the system queries the metadata table to see if the cached data associated with the inference request exists in the storage module 13.

[0051] In some embodiments, the metadata table is a multi-level KVCache metadata table. The global scheduling module 11 retrieves inference requests in the request queue, matches and obtains all cached data blocks (i.e., KVCacheBlocks) associated with the inference request. The association is determined based on the correspondence between the prompt words or prefix features in the inference request and the content features of the cached data blocks.

[0052] If it is determined that there is cached data associated with the inference request in the storage module 13, the load information of the at least one inference module 12 is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate.

[0053] In some embodiments, the load information includes, but is not limited to, the number of requests being processed by each inference module 12, the number of queued requests, and the cache data utilization rate (KVCache utilization). The global scheduling module 11 obtains all load information in real time to comprehensively reflect the current processing capacity of the inference module.

[0054] The transmission cost of transferring the cached data from the storage module 13 to the at least one inference module 12 is compared with the computation cost of the at least one inference module 12 to obtain the comparison result.

[0055] In some embodiments, the transmission cost is the transmission cost of transmitting the matched cached data block to each inference module 12. The transmission cost includes, but is not limited to, quantitative indicators such as loading time, communication bandwidth, and link quality. The computation cost is the time cost for the inference module 12 to perform sequence filling (Prefill) calculation to generate the corresponding cached data for the inference request. The global scheduling module 11 directly compares the two to determine the efficiency priority.

[0056] Based on the load information and the comparison results, the target inference module is determined from the at least one inference module 12.

[0057] In some embodiments, the global scheduling module 11 first filters out lightly loaded active inference modules 12 based on load information, and then selects the optimal lightly loaded inference module as the target inference module by combining the comparison results of transmission cost and computing cost, and instructs the cache client of the target inference module to subscribe to or pull the matching cache data block; if there is no lightly loaded inference module 12, the global scheduling module 11 calls the relevant interface to start a new inference module 12 with the lowest transmission cost of matching cache data block, and completes the preloading of the cache data block.

[0058] The above structure enables precise reuse of cached data, load balancing and elastic scaling of the inference module, reduces the overhead of redundant calculations and cache data transmission, and ensures the overall throughput efficiency of the inference system.

[0059] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: If the duration of the preset load threshold of the at least one inference module 12 is greater than the preset time threshold, after the current inference request is processed, new inference requests are rejected and the module enters a sleep mode.

[0060] In some embodiments, the preset load threshold is a configurable specific threshold N, and the preset time threshold is a configurable specific time T. When the load of the active inference module 12 is lower than the threshold N and the duration exceeds T, the inference module 12 sends a request to the global scheduling module 11. After completing the inference request currently being processed, it no longer receives new requests and enters a sleep mode.

[0061] The above structure avoids resource waste caused by long-term idleness of the inference module and optimizes the dynamic allocation efficiency of system resources.

[0062] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: If it is determined that the target inference module has failed while processing the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost.

[0063] In some embodiments, the fault includes abnormal situations such as sudden downtime during the request processing of the inference module. After the global scheduling module 11 detects the fault, it creates a new inference module 12 according to the logic of load information evaluation and transmission cost calculation through an elastic scaling mechanism.

[0064] The cached data generated in the faulty inference module that has failed, as well as the target cached data corresponding to the inference request, are migrated to the new inference module through a preloading mechanism.

[0065] In some embodiments, the preloading mechanism is a KVCache preloading mechanism, through which the global scheduling module 11 loads the cached data generated by the fault inference module and the corresponding target cached data from the storage module 13 to the newly created inference module 12.

[0066] The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

[0067] In some embodiments, the global scheduling module 11 redistributes the inference requests originally allocated to the fault inference module, as well as the partial text results generated by the fault inference module before the fault occurred, to the new inference module 12. The new inference module 12 continues the incomplete inference tasks based on the preloaded cached data until a complete result is generated and fed back.

[0068] The above structure enables fault hot migration of inference tasks, which is seamless for users and ensures the continuity and stability of inference services.

[0069] Please continue reading. Figure 1 The inference module 12 is also used for: During the sequence filling stage of model inference, the cache data generated by each layer of the model is transferred to the host memory of the storage module 13.

[0070] In some embodiments, the sequence filling stage is the Prefill stage, in which the inference module 12 transmits the cached data generated by each layer of the model to the page-locked CPU memory (i.e., host memory) of the storage module 13 in real time.

[0071] During the decoding and generation phase of model inference, at least one cached data required for the current computation is saved to the processor's video memory, and the cached data not used for computation is saved in the host memory; wherein, the inference module 12 is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

[0072] In some embodiments, during the decoding generation stage, the inference module 12 only stores a portion of the cached data required for the current computation in the processor's video memory (i.e., GPU video memory), while the remaining unused cached data is kept in the host memory to be loaded. Furthermore, the timing configuration ensures that the computational operations of each layer overlap with the loading communication operations of the next layer's cached data, thereby masking the loading overhead.

[0073] Through the above structure, the layer-affinity cache data management and offloading mechanism reduces the processor's memory usage and avoids memory exhaustion during long context inference.

[0074] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: The at least one inference module 12 is divided into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests.

[0075] In some embodiments, long sequence inference requests are inference requests corresponding to long context prompts. The global scheduling module 11 divides multiple inference modules 12 into a parallel processing group specifically for processing this type of long sequence inference request.

[0076] During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

[0077] In some embodiments, during each round of iterative processing, the global scheduling module 11 dynamically adjusts the number of inference modules 12 participating in the parallel processing group based on the current round's demand for computing resources and the memory usage, i.e., changes the degree of parallelism, in order to reduce the cache data memory usage of a single inference module 12.

[0078] Through the above structure, namely the dynamic adjustment mechanism of parallelism of sequence parallelism, the resource requirements of long sequence inference requests at different processing stages are adapted, thereby improving the throughput of single modules and the system as a whole.

[0079] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: In the sequence filling stage for processing the long sequence inference request, all inference modules 12 in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result.

[0080] In some embodiments, for the sequence filling stage (Prefill stage) of a long sequence inference request, because the computational power consumption of this stage is large, the global scheduling module 11 controls all the inference modules 12 in the parallel processing group to jointly process the request in parallel and generate the primary processing result (i.e., the corresponding cached data).

[0081] In the decoding generation stage of processing the primary processing results, the number of inference modules 12 is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0082] In some embodiments, after a long sequence inference request enters the decoding generation stage (Decode stage), the initial rounds consume less computing power and can be processed by only one inference module 12 in the parallel processing group, while the remaining modules can handle the sequence filling stage tasks of other inference requests; as the decoding iteration rounds increase, if the computing power requirement increases or the single module's video memory cannot accommodate all the cached data, the number of inference modules 12 participating in the processing of the request will be dynamically increased.

[0083] The above structure enables dynamic allocation of computing power at different processing stages, while also taking into account the parallel processing efficiency of multiple inference requests.

[0084] Please continue reading. Figure 1 The global scheduling module 11 is further configured to: If it is determined that the parallelism of the parallel processing group has decreased, the cached data corresponding to the inference module 12 in the parallel processing group is saved; wherein, the parallelism is the number of inference modules 12.

[0085] In some embodiments, when the parallelism of the parallel processing group (the number of inference modules 12) decreases, the global scheduling module 11 controls the retention of cached data in the video memory of the corresponding inference module 12 in the parallel processing group, that is, selectively reserving cached data.

[0086] If the parallelism of the parallel processing group is increased, the cached data stored in the inference module 12 is sent to the new inference module for reuse.

[0087] In some embodiments, when the parallelism of the parallel processing group increases, the global scheduling module 11 sends the cached data reserved in the original inference module 12 to the newly added inference module 12 for reuse; when the parallel processing group is adjusted from the original parallel group P to the new parallel group P', the cached data of the same inference module 12 in the two groups is selectively retained, so as to realize zero-overhead transfer of cached data in different rounds.

[0088] The above structure further reduces the communication time overhead of cached data during parallelism adjustment, ensuring efficient reuse of cached data.

[0089] To clearly illustrate the embodiments, Figure 2 An architecture diagram of a data storage system provided by an embodiment of this disclosure is shown. For example... Figure 2As shown in the diagram, this system's layered architecture and the functional collaboration logic of its components are illustrated: the top layer is the global scheduling module, responsible for scheduling inference requests, prefetching cached data, fault migration, and elastic scaling; the middle layer is the distributed inference unit of the inference module, which is divided into processing modules corresponding to the sequence filling stage and the decoding generation stage, with each stage module associated with its corresponding inference engine and distributed cached data; the lower layer is the service layer cache management unit of the storage module, which performs multi-level unified management of cached data and performs intelligent routing, preloading, and cached data quantization and insertion algorithms; the bottom layer is the multi-level storage media cluster of the storage module, providing low-latency, high-throughput network transmission support for multiple clusters and nodes. The overall architecture achieves coordinated scheduling and efficient management of inference requests, cached data, and system resources.

[0090] To clearly illustrate the embodiments, Figure 3 This illustration shows a cached data structure and an interaction diagram of an inference module provided in an embodiment of this disclosure. For example... Figure 3 As shown in the figure, the diagram presents two parts: the left side shows the structure of cached data in the storage module, where cached data exists in the form of key vectors and value vectors, containing dimensional information such as the number of channels, the number of layers (vertical and horizontal), and the number of tokens; the right side shows the interaction logic between the global scheduling module and at least one inference module (inference module 1, ..., inference module n) and the storage module, that is, the global scheduling module is responsible for coordinating the access of cached data between the inference module and the storage module, and can also schedule the transmission and reuse of cached data between different inference modules.

[0091] To clearly illustrate the embodiments, Figure 4 A schematic diagram illustrating a cached data transmission parameter provided in an embodiment of this disclosure is shown. For example... Figure 4As shown, the figure illustrates two parts: the left side presents the cache data transmission bandwidth parameters for different storage media in the storage module (corresponding to the destination device, including GPU memory, host memory (paged CPU memory and paged-locked CPU memory), local storage device (local SSD), and remote storage server); the right side presents the operations of the inference module during the prefill phase, including prefilling operations on GPU memory, loading cache data from paged-locked CPU memory, loading cache data from local SSD, loading cache data from remote storage server, etc., as well as the transmission bandwidth and time (i.e., transmission cost) corresponding to the operations of loading cache data from different storage media in the storage module. This clarifies the differences in resource consumption for cache data transmission under different storage media, providing parameter basis for cache data storage strategies and loading scheduling.

[0092] To clearly illustrate the embodiments, Figure 5 This illustration shows a logic diagram of a cross-inference module provided in an embodiment of the present disclosure, such as... Figure 5As shown in the diagram, this diagram illustrates the cache data interaction and categorized storage logic among multiple inference modules: "Instance 1, Instance 2, Instance 3" on the left correspond to different inference modules. When an inference request flows between instances, the "KVCache Management" unit within each inference module synchronously processes the cache data (KVCache) corresponding to that request. For example, when an inference request flows from Instance 1 to Instance 2, the KVCache Management unit of Instance 1 will transfer the corresponding KVCache to the KVCache Management unit of Instance 2, ensuring cache reuse during the inference process. The right side shows the hierarchical categorized storage structure of cache data blocks, where each identifier is defined according to rules ("P" represents the Prefill sequence filling stage, "D" represents the Decode decoding generation stage, and "B" represents a cache block), including PB211 (the first cache data block generated by Instance 2 in the first round of the Prefill stage), DB2... DB233 (the first cached data generated in the first round of the Decode phase for instance 2), DB232 (the second cached data generated in the third round of the Decode phase for instance 2), PB311 (the first cached data generated in the first round of the Prefill phase for instance 3), DB111 (the first cached data generated in the first round of the Decode phase for instance 1), DB112 (the second cached data generated in the first round of the Decode phase for instance 1), and DB113 (the third cached data generated in the first round of the Decode phase for instance 1) contain round-based identifiers and hierarchical storage logic. This allows the global scheduling module to accurately locate cached data in different stages, different inference modules, and different rounds, further improving the accuracy and efficiency of cache reuse and cross-module scheduling.

[0093] Corresponding to the aforementioned data storage system, this invention also proposes a data storage method. Since the method embodiments of this invention correspond to the aforementioned system embodiments, details not disclosed in the method embodiments can be referred to the aforementioned system embodiments, and will not be repeated here.

[0094] Figure 6 This is a schematic flowchart illustrating a data storage method provided in an embodiment of the present disclosure.

[0095] like Figure 6 As shown, the method includes the following steps: Step 101: In response to receiving an inference request, determine whether there is cached data in the storage module associated with the inference request based on the inference request.

[0096] In some embodiments, the inference request is a task request that needs to be processed by a preset language model, including but not limited to text generation, question-and-answer interaction, etc.; the cached data is KVCache data, that is, the matrix data corresponding to the key vector and value vector generated during the inference process; the determination method is: query the multi-level KVCache metadata table in the storage module, extract the prompt words or prefix features in the inference request, match them with the content features of the cached data block (KVCacheBlock) in the metadata table, and retrieve all associated cached data blocks.

[0097] The above methods enable accurate retrieval of cached data, providing a data foundation for cache reuse.

[0098] Step 102: If it is determined that there is cached data associated with the inference request, obtain the load information of at least one inference module and the transmission cost of the cached data to the at least one inference module.

[0099] In some embodiments, load information includes the number of requests being processed by each inference module, the number of queued requests, batch size, and cache data utilization (KVCache utilization); transmission cost is the cost of transmitting cached data blocks to each inference module, including quantitative indicators such as loading time, communication bandwidth, and link quality; load information is obtained by real-time sensing of the status of each active inference module, and transmission cost is determined by calculating the transmission time of cached data blocks to each inference module.

[0100] The above methods comprehensively obtained information on the operating status and data transmission of the inference module, providing a complete basis for the selection of the target inference module.

[0101] Step 103: Based on the load information and the transmission cost, determine the target inference module from the at least one inference module.

[0102] In some embodiments, lightly loaded active inference modules are first selected based on load information. Then, the cost of transmitting cached data is compared with the cost of performing sequence prefill calculation in the inference module, and the lightly loaded inference module with the best overall performance is selected as the target inference module. If no lightly loaded inference module exists, a new inference module with the lowest cost of transmitting cached data blocks is started, and cached data preloading is completed.

[0103] The above method achieves load balancing and elastic scaling of the inference module, reducing redundant calculations and transmission overhead.

[0104] Step 104: Schedule the inference request to the target inference module.

[0105] In some embodiments, the target inference module's cache client is synchronously instructed to subscribe to or pull matching cache data blocks during scheduling, ensuring that the target inference module can directly reuse cache data for inference after receiving an inference request.

[0106] The above method enables coordinated scheduling of inference requests and cached data, ensuring efficient execution of the inference process.

[0107] In summary, the data storage method provided in this disclosure includes: responding to receiving an inference request, determining whether cached data associated with the inference request exists in the storage module based on the inference request; if it is determined that cached data associated with the inference request exists, obtaining load information of at least one inference module and obtaining the transmission cost of the cached data to at least one inference module; determining a target inference module from the at least one inference module based on the load information and the transmission cost; scheduling the inference request to the target inference module; and improving the overall utilization rate of cached resources and the performance of inference services by optimizing the resource management and scheduling mechanism of cached data, thereby enhancing the concurrent carrying capacity and response efficiency of the inference service.

[0108] Figure 7 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 7 As shown, the method includes: Step 201: In response to receiving the inference request, if it is determined in the metadata table that there is cached data associated with the inference request in the storage module, the load information of the at least one inference module is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate.

[0109] Step 202: Compare the transmission cost of the cached data from the storage module to the at least one inference module with the computation cost of the at least one inference module to obtain a comparison result.

[0110] Step 203: Based on the load information and the comparison results, determine the target inference module from the at least one inference module.

[0111] In some embodiments of this disclosure, the method further includes: if the duration of a preset load threshold of the at least one inference module is greater than a preset time threshold, after processing the currently processed inference request, rejecting new inference requests and entering a sleep mode.

[0112] Figure 8 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 8 As shown, the method includes: Step 301: If it is determined that the target inference module has failed during the processing of the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost.

[0113] Step 302: The cached data generated in the faulty inference module that has failed and the target cached data corresponding to the inference request are migrated to the new inference module through a preloading mechanism.

[0114] Step 303: The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

[0115] Figure 9 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 9 As shown, the method includes: Step 401: Divide the at least one inference module into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests.

[0116] Step 402: During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

[0117] Figure 10 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 10 As shown, the method includes: Step 501: In the sequence filling stage for processing the long sequence inference request, all inference modules in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result.

[0118] Step 502: In the decoding generation stage of processing the primary processing results, the number of inference modules is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0119] Figure 11 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 11 As shown, the method includes: Step 601: If it is determined that the parallelism of the parallel processing group has decreased, save the cached data corresponding to the inference module in the parallel processing group; wherein, the parallelism is the number of inference modules.

[0120] Step 602: If it is determined that the parallelism of the parallel processing group has increased, the cached data stored in the inference module is sent to the new inference module for reuse.

[0121] Figure 12 This is a schematic flowchart illustrating a data storage method provided in an embodiment of the present disclosure.

[0122] like Figure 12 As shown, the method includes the following steps: Step 701: In response to receiving an inference request, load the corresponding cached data from the storage module based on the inference request.

[0123] In some embodiments, inference requests are distributed to the inference module by the global scheduling module. Upon receiving the request, the inference module extracts the cached data identifier (associated with the prompt word features, model identifier, etc. of the inference request) and sends a load request to the storage module. The storage module queries the multi-level KVCache metadata table based on this identifier to locate the corresponding cached data block (KVCacheBlock) and transmits the data to the inference module via direct memory access or network transmission.

[0124] The above method enables accurate loading of cached data, providing data support for reusing historical calculation results in the inference process.

[0125] Step 702: Perform model inference based on the cached data.

[0126] In some embodiments, a preset language model is deployed inside the inference module. The model inference is divided into a sequence filling stage (Prefill) and a decoding generation stage (Decode). In the sequence filling stage, the model uses the loaded cached data to process historical tokens, and the newly generated cached data at each layer is transmitted to the page-locked CPU memory of the storage module in real time. In the decoding generation stage, only a portion of the cached data required for the current calculation is stored in the processor's video memory, and the rest is stored in the host memory. The calculation operation at each layer overlaps with the loading operation of the cached data at the next layer.

[0127] The above method avoids the repeated calculation of historical tokens, reduces processor memory usage, and improves inference efficiency.

[0128] Step 703: Send the target cache data generated during the model inference process to the storage module so that the storage module can store it and reuse the inference request.

[0129] In some embodiments, the target cache data is a matrix of key vectors and value vectors added during the inference process. After the inference module adds an association identifier (including inference request ID, token sequence range, and model hierarchical information) to it, it is sent to the storage module. The storage module classifies and stores it according to a preset multi-level logical structure and updates the metadata table for retrieval and reuse by associated inference requests.

[0130] The above methods enable the continuous accumulation and cyclical reuse of cached data, laying the foundation for efficient processing of inference requests.

[0131] In summary, the data storage method disclosed herein includes: in response to receiving an inference request, loading corresponding cached data from a storage module based on the inference request; performing model inference based on the cached data; sending target cached data generated during the model inference process to the storage module so that the storage module can store and reuse the inference request; by optimizing the resource management and scheduling mechanism of cached data, the overall utilization rate of cached resources and the performance of inference services are improved, and the concurrent carrying capacity and response efficiency of inference services are enhanced.

[0132] Figure 13 A flowchart illustrating a data storage method provided in an embodiment of this disclosure is further shown. For example... Figure 13 As shown, the method includes: Step 801: In the sequence filling stage of model inference, the cache data generated by each layer of the model calculation is transferred to the host memory of the storage module.

[0133] Step 802: In the decoding and generation stage of model inference, at least one cached data currently being computed is saved to the processor's video memory, and the cached data not yet computed is saved in the host memory; wherein, the inference module is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

[0134] Corresponding to the data storage method described above, this invention also proposes a data storage device. Since the device embodiments of this invention correspond to the method embodiments described above, details not disclosed in the device embodiments can be referred to in the method embodiments described above, and will not be repeated here.

[0135] Figure 14 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present disclosure, such as... Figure 14 As shown, it includes: a first determining unit 51, an acquiring unit 52, a second determining unit 53, and a scheduling unit 54.

[0136] The first determining unit 51 is configured to, in response to receiving an inference request, determine whether there is cached data associated with the inference request in the storage module based on the inference request; The acquisition unit 52 is used to acquire the load information of at least one inference module and the transmission cost from the cached data to the at least one inference module when it is determined that there is cached data associated with the inference request. The second determining unit 53 is used to determine a target inference module from the at least one inference module based on the load information and the transmission cost; The scheduling unit 54 is used to schedule the inference request to the target inference module.

[0137] In summary, the data storage device provided in this disclosure includes: responding to receiving an inference request, determining whether cached data associated with the inference request exists in a storage module based on the inference request; if it is determined that cached data associated with the inference request exists, obtaining load information of at least one inference module and obtaining the transmission cost of the cached data to at least one inference module; determining a target inference module from the at least one inference module based on the load information and the transmission cost; scheduling the inference request to the target inference module; and improving the overall utilization rate of cached resources and the performance of inference services by optimizing the resource management and scheduling mechanism of cached data, thereby enhancing the concurrent carrying capacity and response efficiency of the inference service.

[0138] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, the second determining unit 53 includes: The acquisition module 531 is configured to, in response to receiving the inference request, and if it is determined in the metadata table that the cached data associated with the inference request exists in the storage module, acquire the load information of the at least one inference module; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; The comparison module 532 is used to compare the transmission cost of the cached data from the storage module to the at least one inference module with the computation cost of the at least one inference module, and obtain a comparison result. The determination module 533 is used to determine the target inference module from the at least one inference module based on the load information and the comparison result.

[0139] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, the device further includes: The switching unit 55 is configured to, after the scheduling unit 54 schedules the inference request to the target inference module, and if it is determined that the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after processing the currently processed inference request, refuse to receive new inference requests and enter a sleep mode.

[0140] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, the device further includes: The creation unit 56 is used to create a new inference module based on the load information and the transmission cost, after the scheduling unit 54 schedules the inference request to the target inference module and determines that the target inference module has failed in the process of processing the inference request. Migration unit 57 is used to migrate the cached data generated in the faulty inference module that has failed and the target cached data corresponding to the inference request to the new inference module through a preloading mechanism. The first sending unit 58 is used to resend the inference request and a portion of the text result generated by the fault inference module to the new inference module to continue completing the inference request.

[0141] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, the scheduling unit 54 includes: The partitioning module 541 is used to divide the at least one inference module into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; The adjustment module 542 is used to dynamically adjust the number of inference modules being processed in the current round of the parallel processing group according to the computing power resources and memory usage requirements during the long sequence inference request processing.

[0142] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, the adjustment module 542 includes: The processing submodule 5421 is used to process the long sequence inference request in parallel using all the inference modules in the parallel processing group during the sequence filling stage of processing the long sequence inference request, so as to obtain the initial processing result. A submodule 5422 is added to dynamically increase the number of inference modules during the decoding generation stage of processing the primary processing results, based on the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

[0143] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 15 As shown, it includes: The first storage module 543 is configured to, after the partitioning module 541 has divided the at least one inference module into a parallel processing group, and when it is determined that the parallelism of the parallel processing group has decreased, store the cached data corresponding to the inference module in the parallel processing group; wherein, the parallelism is the number of inference modules; The sending module 544 is used to send the cached data stored in the inference module to a new inference module for reuse when it is determined that the parallelism of the parallel processing group has increased.

[0144] Figure 16 This is a schematic diagram of the structure of a data storage device provided in an embodiment of the present disclosure, such as... Figure 16 As shown, it includes: a loading unit 61, an inference unit 62, and a second sending unit 63.

[0145] Loading unit 61 is used to load corresponding cached data from the storage module based on the received inference request. Inference unit 62 is used to perform model inference based on the cached data; The second sending unit 63 is used to send the target cache data generated during the model inference process to the storage module so that the storage module can store it and reuse the inference request.

[0146] In summary, the data storage device according to this disclosure includes: in response to receiving an inference request, loading corresponding cached data from a storage module based on the inference request; performing model inference based on the cached data; sending target cached data generated during the model inference process to the storage module so that the storage module can store and reuse the inference request; by optimizing the resource management and scheduling mechanism of cached data, the overall utilization rate of cached resources and the performance of inference services are improved, and the concurrent carrying capacity and response efficiency of inference services are enhanced.

[0147] Furthermore, in one possible implementation of the embodiments of this disclosure, such as Figure 17 As shown, the reasoning unit 62 includes: The transmission module 621 is used to transmit the cache data generated by each layer of the model to the host memory of the storage module during the sequence filling stage of model inference. The second storage module 622 is used to save at least one cached data currently being computed to the processor's video memory and save uncomputed cached data to the host memory during the decoding and generation phase of model inference; wherein the inference module is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

[0148] It should be noted that the foregoing explanation of the method embodiments also applies to the apparatus of the embodiments of this disclosure, and the principle is the same. Therefore, the embodiments of this disclosure are not limited thereto.

[0149] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0150] Figure 18 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0151] like Figure 18 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM (Read-Only Memory) 602 or loaded from storage unit 608 into RAM (Random Access Memory) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An I / O (Input / Output) interface 605 is also connected to the bus 604.

[0152] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, CPUs (Central Processing Units), GPUs (Graphics Processing Units), various special-purpose AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, DSPs (Digital Signal Processors), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as data storage methods. For example, in some embodiments, the data storage method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the aforementioned data storage method by any other suitable means (e.g., by means of firmware).

[0154] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application-Specific Standard Products), SOCs (System-on-Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include LANs (Local Area Networks), WANs (Wide Area Networks), the Internet, and blockchain networks.

[0159] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0160] It's important to note that artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0161] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0162] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data storage system, characterized in that, The system includes: a global scheduling module, at least one inference module, and a storage module, wherein... The global scheduling module is used to determine a target inference module from the at least one inference module based on the load information of the at least one inference module and the transmission cost of obtaining the cached data to the at least one inference module when it is determined that there is cached data associated with the received inference request in the storage module, and to schedule the inference request to the target inference module. The at least one inference module is configured to respond to receiving the inference request sent by the global scheduling module, load the cached data corresponding to the inference request from the storage module based on the inference request, perform model inference based on the cached data, and send the target cached data generated during the model inference process to the storage module; The storage module is used to store the target cache data sent by the inference module, and manage the cache data and the target cache data according to a predefined structure, so as to provide the cache data access service to the at least one inference module.

2. The system according to claim 1, characterized in that, The global scheduling module is also used for: In response to receiving the inference request, the system queries the metadata table to determine whether the cached data associated with the inference request exists in the storage module. If it is determined that cached data associated with the inference request exists in the storage module, the load information of the at least one inference module is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; The transmission cost of transferring the cached data from the storage module to the at least one inference module is compared with the computation cost of the at least one inference module to obtain the comparison result; Based on the load information and the comparison results, the target inference module is determined from the at least one inference module.

3. The system according to claim 1, characterized in that, The global scheduling module is also used for: If the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after the current inference request is processed, new inference requests are rejected and the module enters a sleep mode.

4. The system according to claim 1, characterized in that, The global scheduling module is also used for: If it is determined that the target inference module has failed while processing the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost. The cached data generated in the faulty inference module that has failed, as well as the target cached data corresponding to the inference request, are migrated to the new inference module through a preloading mechanism. The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

5. The system according to claim 1, characterized in that, The inference module is also used for: During the sequence filling stage of model inference, the cache data generated by each layer of the model is transferred to the host memory of the storage module; During the decoding and generation phase of model inference, at least one cached data required for the current computation is saved to the processor's video memory, and the cached data not yet used for computation is saved in the host memory; wherein, the inference module is configured to make the computation operation of each layer overlap with the communication operation of loading the cached data required for the next layer from the host memory in time.

6. The system according to claim 1, characterized in that, The global scheduling module is also used for: The at least one inference module is divided into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

7. The system according to claim 6, characterized in that, The global scheduling module is also used for: In the sequence filling stage of processing the long sequence inference request, all inference modules in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result; In the decoding and generation stage of processing the initial processing results, the number of inference modules is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

8. The system according to claim 6 or 7, characterized in that, The global scheduling module is also used for: If it is determined that the parallelism of the parallel processing group has decreased, the cached data corresponding to the inference module in the parallel processing group is saved; wherein, the parallelism is the number of inference modules; If the parallelism of the parallel processing group is increased, the cached data stored in the inference module is sent to the new inference module for reuse.

9. A data storage method, characterized in that, include: In response to receiving an inference request, determine whether there is cached data in the storage module associated with the inference request based on the inference request; If it is determined that there is cached data associated with the inference request, obtain the load information of at least one inference module and the transmission cost of the cached data to the at least one inference module; Based on the load information and the transmission cost, a target inference module is determined from the at least one inference module; The inference request is scheduled to the target inference module.

10. The method according to claim 9, characterized in that, The step of determining the target inference module from the at least one inference module based on the load information and the transmission cost includes: In response to receiving the inference request, if it is determined in the metadata table that there is cached data associated with the inference request in the storage module, the load information of the at least one inference module is obtained; wherein, the load information includes the number of requests being processed by each inference module, the number of queued requests, and the cache data occupancy rate; The transmission cost of transferring the cached data from the storage module to the at least one inference module is compared with the computation cost of the at least one inference module to obtain the comparison result; Based on the load information and the comparison results, the target inference module is determined from the at least one inference module.

11. The method according to claim 9, characterized in that, After scheduling the inference request to the target inference module, the method further includes: If the duration of the preset load threshold of the at least one inference module is greater than the preset time threshold, after the current inference request is processed, new inference requests are rejected and the module enters a sleep mode.

12. The method according to claim 9, characterized in that, After scheduling the inference request to the target inference module, the method further includes: If it is determined that the target inference module has failed while processing the inference request, a new inference module is created through an elastic scaling mechanism based on the load information and the transmission cost. The cached data generated in the faulty inference module that has failed, as well as the target cached data corresponding to the inference request, are migrated to the new inference module through a preloading mechanism. The inference request and the partial text results generated by the fault inference module are resent to the new inference module to continue completing the inference request.

13. The method according to claim 9, characterized in that, The step of scheduling the inference request to the target inference module includes: The at least one inference module is divided into a parallel processing group; wherein the parallel processing group is used to process long sequence inference requests; During the long sequence inference request processing, the number of inference modules being processed in the current round in the parallel processing group is dynamically adjusted according to the computing power resources and memory usage requirements.

14. The method according to claim 13, characterized in that, During the long-sequence inference request processing, the number of inference modules being processed in the current round of the parallel processing group is dynamically adjusted according to computing resources and memory usage requirements, including: In the sequence filling stage of processing the long sequence inference request, all inference modules in the parallel processing group are used to process the long sequence inference request in parallel to obtain the initial processing result; In the decoding and generation stage of processing the initial processing results, the number of inference modules is dynamically increased according to the increase in computing power resources in the current round or the saturation of the video memory usage requirements.

15. The method according to claim 13 or 14, characterized in that, After dividing the at least one inference module into a parallel processing group, the method further includes: If it is determined that the parallelism of the parallel processing group has decreased, the cached data corresponding to the inference module in the parallel processing group is saved; wherein, the parallelism is the number of inference modules; If the parallelism of the parallel processing group is increased, the cached data stored in the inference module is sent to the new inference module for reuse.

16. A data storage method, characterized in that, include: In response to receiving an inference request, the corresponding cached data is loaded from the storage module based on the inference request; Model inference is performed based on the cached data; The target cache data generated during the model inference process is sent to the storage module so that the storage module can store it and reuse the inference request.

17. The method according to claim 16, characterized in that, The model inference based on the cached data includes: During the sequence filling stage of model inference, the cache data generated by each layer of the model is transferred to the host memory of the storage module; During the decoding and generation phase of model inference, at least one cached data required for the current computation is saved to the processor's video memory, while uncomputed cached data is saved in the host memory. The inference module is configured to overlap the computation operations of each layer with the communication operations that load the cached data required for the next layer from the host memory in time.

18. A data storage device, characterized in that, include: The first determining unit is configured to, in response to receiving an inference request, determine whether there is cached data associated with the inference request in the storage module based on the inference request; The acquisition unit is configured to, upon determining the existence of cached data associated with the inference request, acquire load information of at least one inference module and acquire the transmission cost of the cached data to the at least one inference module; The second determining unit is used to determine a target inference module from the at least one inference module based on the load information and the transmission cost; The scheduling unit is used to schedule the inference request to the target inference module.

19. A data storage device, characterized in that, include: A loading unit is used to load corresponding cached data from the storage module based on the received inference request. The inference unit is used to perform model inference based on the cached data; The second sending unit is used to send the target cache data generated during the model inference process to the storage module so that the storage module can store it and reuse the inference request.

20. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 9-15 or 16-17.

21. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 9-15 or 16-17.

22. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 9-15 or 16-17.

Images